A new method of evaluating the accuracy of economic forecasts

Conventional measures of forecasting accuracy reflect the view that forecast evaluation should concentrate on all large disturbances and ignore turning-point errors. Many forecasters, however, believe missed turns are the most grievous of all forecasting errors. Despite this consensus, no generally acceptable measure of this type of forecasting error exists. In this paper, such a measure—the probability of correctly forecasting directional change—is introduced. Values of this measure are computed for eleven well-known macroeconometric forecasting models. An inequality-type index of relative directional accuracy based on this measure is also presented and used to evaluate the models in terms of their relative accuracy.     

Does the vector error correction model perform better than others in forecasting stock price? An application of residual income valuation theory

This paper employs a multi-equation model approach to consider three statistic problems (heteroskedasticity, endogeneity and persistency), which are sources of bias and inefficiency in the predictive regression models. This paper applied the residual income valuation model (RIM) proposed by Ohlson (1995) to forecast stock prices for Taiwan three sectors. We compare relative forecasting accuracy of vector error correction model (VECM) with the vector autoregressive model (VAR) as well as OLS and RW models used in the prior studies. We conduct out-of-sample forecasting and employ two instruments to assess forecasting performance. Our empirical results suggest that the VECM statistically outperforms other three models in forecasting stock prices. When forecasting horizons extend longer, VECM produces smaller forecasting errors and performs substantially better than VAR, suggesting that the ability of VECM to improve VAR forecast accuracy is stronger with longer horizons. These findings imply that an error correction term (ECT) of the VECM contributes to improving forecast accuracy of stock prices. Our economic significance analyses and robustness tests for different data frequency are in favor of the superiority of VECM estimator.     

Combination forecasts of tourism demand with machine learning models

The main objective of this study is to analyse whether the combination of regional predictions generated with machine learning (ML) models leads to improved forecast accuracy. With this aim, we construct one set of forecasts by estimating models on the aggregate series, another set by using the same models to forecast the individual series prior to aggregation, and then we compare the accuracy of both approaches. We use three ML techniques: support vector regression, Gaussian process regression and neural network models. We use an autoregressive moving average model as a benchmark. We find that ML methods improve their forecasting performance with respect to the benchmark as forecast horizons increase, suggesting the suitability of these techniques for mid- and long-term forecasting. In spite of the fact that the disaggregated approach yields more accurate predictions, the improvement over the benchmark occurs for shorter forecast horizons with the direct approach.     

Forecasting economic activity by Bayesian bridge model averaging

This paper proposes the use of Bayesian model averaging (BMA) as an alternative tool to forecast GDP relative to simple bridge models and factor models. BMA is a computationally feasible method that allows us to explore the model space even in the presence of a large set of candidate predictors. We test the performance of BMA in now-casting by means of a recursive experiment for the euro area and the three largest countries. This method allows flexibility in selecting the information set month by month. We find that BMA-based forecasts produce smaller forecast errors than standard bridge model when forecasting GDP in Germany, France and Italy. At the same time, it also performs as well as medium-scale factor models when forecasting Eurozone GDP.     

On winning forecasting competitions in economics

To explain which methods might win forecasting competitions on economic time series, we consider forecasting in an evolving economy subject to structural breaks, using mis-specified, data-based models. ‘Causal’ models need not win when facing deterministic shifts, a primary factor underlying systematic forecast failure. We derive conditional forecast biases and unconditional (asymptotic) variances to show that when the forecast evaluation sample includes sub-periods following breaks, non-causal models will outperform at short horizons. This suggests using techniques which avoid systematic forecasting errors, including improved intercept corrections. An application to a small monetary model of the UK illustrates the theory.     

Comparative accuracy of forecasts of inflation: a Canadian study

An inaccurate forecast of inflation is costlier to economic agents when the inflation rate is high and volatile. In this situation, the use of more sophisticated and information-oriented forecasting models become economically efficient. We test this hypothesis by analysing the forecasting accuracy of vector auto-regressive (VAR), auto-regressive integrated moving average (ARIMA) and static expectation models. We use Canadian data and divide the post-sample forecasting period into four sub-periods, based on high/low and volatile/stable inflation. Prediction errors are compared for both short-term and long-term forecasts. Finally, the paper proposes a portfolio approach for obtaining a more accurate forecast of inflation.     

UK macroeconomic forecasting with many predictors: Which models forecast best and when do they do so?

Block factor methods offer an attractive approach to forecasting with many predictors. These extract the information in these predictors into factors reflecting different blocks of variables (e.g. a price block, a housing block, a financial block, etc.). However, a forecasting model which simply includes all blocks as predictors risks being over-parameterized. Thus, it is desirable to use a methodology which allows for different parsimonious forecasting models to hold at different points in time. In this paper, we use dynamic model averaging and dynamic model selection to achieve this goal. These methods automatically alter the weights attached to different forecasting models as evidence comes in about which has forecast well in the recent past. In an empirical study involving forecasting output growth and inflation using 139 UK monthly time series variables, we find that the set of predictors changes substantially over time. Furthermore, our results show that dynamic model averaging and model selection can greatly improve forecast performance relative to traditional forecasting methods.     

Performance of periodic time series models in forecasting

The paper provides a comparison of alternative univariate time series models that are advocated for the analysis of seasonal data. Consumption and income series from (West-) Germany, United Kingdom, Japan and Sweden are investigated. The performance of competing models in forecasting is used to assess the adequacy of a specific model. To account for nonstationarity first and annual differences of the series are investigated. In addition, time series models assuming periodic integration are evaluated. To describe the stationary dynamics (standard) time invariant parametrizations are compared with periodic time series models conditioning the data generating process on the season. Periodic models improve the in-sample fit considerably but in most cases under study this model class involves a loss in ex-ante forecasting relative to nonperiodic models. Inference on unit-roots indicates that the nonstationary characteristics of consumption and income data may differ. For German and Swedish data forecasting exercises yield a unique recommendation of unit roots in consumption and income data which is an important (initial) result for multivariate analysis. Time series models assuming periodic integration are parsimonious to specify but often involve correlated one-step-ahead forecast errors. First version received: April 1996/final version received: January 1998     

A Nonlinear Approach for Modeling and Forecasting US Business Cycles

The purpose of this paper is to provide a complete evaluation of four regime-switching models by checking their performance in detecting US business cycle turning points, in replicating US business cycle features and in forecasting US GDP growth rate. Both individual and combined forecasts are considered. Results indicate that while the Markov-switching model succeeded in replicating all the NBER peak and trough dates without an extra-cycle detection, it seems to be outperformed by the Bounce-back model in term of the delay time to a correct alarm. Concerning business cycle features characterization, none of the competing models dominates over all the features. The performance of the Markov-switching and bounce back models in detecting turning points was not translated into an improved business cycle feature characterization since they are outperformed by the Floor and Ceiling model. The forecast performance of the considered models varies across regimes and across forecast horizons. That is, the model performing best in an expansion period is not necessarily the same in a recession period and similarly for the forecast horizons. Finally, combining such individual forecasts generally leads to increased forecast accuracy especially for h=1.     

Measures of underlying inflation in the euro area: assessment and role for informing monetary policy

The paper evaluates the 24-month-ahead inflation forecasting performance of various indicators of underlying inflation and structural models. Measures derived using the generalized dynamic factor model (GDFM) overperform other measures over the monetary policy horizon and are leading indicators of headline inflation. Trimmed means, although weaker than GDFM indicators, have good forecasting performance, while indicators by permanent exclusion underperform but provide useful information about short-term dynamics. The forecasting performance of theoretically-founded models that relate monetary aggregates, the output gap, and inflation improves with the time horizon but generally falls short of that of the GDFM. A composite measure of underlying inflation, derived by averaging the statistical indicators and the model-based estimates, improves forecast accuracy by eliminating bias and offers valuable insight about the distribution of risks.     

Men, women, and competition: An experimental test of behavior

This study reports a series of experiments that examine outcomes when agents are able to choose between a payment scheme that rewards based on absolute performance (i.e., piece rate) and a scheme that rewards based on relative performance (i.e., a tournament). We test for the presence and persistence of gender differences in performance and the rate of entry into the tournament option and whether these differences are sensitive to the structure of the tournament rewards. In the winner-take-all (WTA) condition, only the best performer in the tournament for each round received a payment ($4.50). In the graduated tournament condition, the same payment ($4.50) was divided among the first, second, and third finishers in the tournament. In the WTA condition, men showed significantly lower forecast errors than women. In addition, a clear sorting effect occurs in the WTA condition. In early rounds of the WTA condition, male entrants into the tournament show significantly lower forecast errors than female tournament entrants. However, the difference disappears over time. After controlling for forecasting skill, gender did not predict entry into the tournament for the WTA condition. However, lower forecasting skill reduced the probability of entry. In the graduated tournament, the situation was reversed. Men entered the tournament at significantly higher rates, even after controlling for skill. Forecasting skill had no impact on the decision to enter the tournament. While the average male entrant to the tournament had lower forecast errors than the average female entrant, the men entered at much higher rates. As a consequence, men were much more likely than women to enter the tournament too frequently.     

Indeterminate forecast accuracy under indeterminacy

This paper studies whether the observed time variation in the forecast accuracy of macro-econometric models can be reconciled with the monetary policy stance that induces (in)determinacy in stylized DSGE models. Using a small-scale New Keynesian monetary framework as laboratory and structural parameters calibrated to the estimates obtained on U.S. data from different macroeconomics regimes, we exploit reduced-form econometric models – such as Vector Autoregressions – to assess their regime-specific forecastability. We show that conducting (pseudo) out-of-sample forecast comparisons in the presence of indeterminacy is a non-trivial exercise, even when sunspot shocks play no role in generating the data. Overall, our simulation experiment suggests that equilibrium indeterminacy need not lead to superior (absolute or relative) forecast accuracy. This finding challenges the view that the deteriorating performance of forecast models over the Great Moderation relative to the Great Inflation was entirely due to changes in the U.S. monetary policy.     

Time-series model forecasts and structural breaks: evidence from Spanish pre-EMU interest rates

Analysis of the future behaviour of economic variables can be biased if structural breaks are not considered. When these structural breaks are present, the in-sample fit of a model gives us a poor guide to ex ante forecast performance. This problem is true for both univariate and multivariate analysis and can be extremely important when co-integration relationships are analysed. The main goal of this article is to analyse the impact of structural breaks on forecast accuracy evaluation. We focus on forecasting several interest rates from the Spanish interbank money market. In order to carry out the analysis, we perform two forecasting exercises: (a) without structural breaks and (b) when structural breaks are explicitly considered. We use new sequential methods in order to estimate change-points in an endogenous way. This method allows us to detect structural breaks in all four rates in May 1993. However, the effects of these breaks are not very strong, since we found scarce gains in forecasting accuracy when the structural breaks are included in the models.     

Forecasting business and consumer surveys indicators–a time-series models competition

The objective of this article is to compare different time-series methods for the short-run forecasting of Business and Consumer Survey Indicators. We consider all available data taken from the Business and Consumer Survey Indicators for the Euro area between 1985 and 2002. The main results of the forecast competition are offered not only for raw data but we also consider the effects of seasonality and removing outliers on forecast accuracy. In most cases, the univariate autoregressions were not outperformed by the other methods. As for the effect of seasonal adjustment methods and the use of data from which outliers have been removed, we obtain that the use of raw data has little effect on forecast accuracy. The forecasting performance of qualitative indicators is important since enlarging the observed time series of these indicators with forecast intervals may help in interpreting and assessing the implications of the current situation and can be used as an input in quantitative forecast models.     

Forecasting house prices using dynamic model averaging approach: Evidence from China

Forecasting house price has been of great interests for macroeconomists, policy makers and investors in recent years. To improve the forecasting accuracy, this paper introduces a dynamic model averaging (DMA) method to forecast the growth rate of house prices in 30 major Chinese cities. The advantage of DMA is that this method allows both the sets of predictors (forecasting models) as well as their coefficients to change over time. Both recursive and rolling forecasting modes are applied to compare the performance of DMA with other traditional forecasting models. Furthermore, a model confidence set (MCS) test is used to statistically evaluate the forecasting efficiency of different models. The empirical results reveal that DMA generally outperforms other models, such as Bayesian model averaging (BMA), information-theoretic model averaging (ITMA) and equal-weighted averaging (EW), in both recursive and rolling forecasting modes. In addition, in recent years it is found that the Google search index, instead of fundamental macroeconomic or monetary indicators, has developed greater predictive power for house price in China.     

A novel time-series model based on empirical mode decomposition for forecasting TAIEX

Stock price prediction is regarded as a challenging task of the financial time series prediction process. Time series models have successfully solved prediction problems in many domains, including the stock market. Unfortunately, there are two major drawbacks in stock market by time-series model: (1) some models cannot be applied to the datasets that do not follow the statistical assumptions; and (2) most time-series models which use stock data with many noises involutedly (caused by changes in market conditions and environments) would reduce the forecasting performance. For solving the above problems and promoting the forecasting performance of time-series models, this paper proposes a hybrid time-series support vector regression (SVR) model based on empirical mode decomposition (EMD) to forecast stock price for Taiwan stock exchange capitalization weighted stock index (TAIEX). In order to evaluate the forecasting performances, the proposed model is compared with autoregressive (AR) model and SVR model. The experimental results show that the proposed model is superior to the listing models in terms of root mean squared error (RMSE). And the more fluctuation year (2000–2001) occurs, the better accuracy of proposed model will be obtained.     

Should Forecasters Use Real‐Time Data to Evaluate Leading Indicator Models for GDP Prediction? German Evidence

In this paper, we investigate whether differences exist among forecasts using real‐time or latest‐available data to predict gross domestic product (GDP). We employ mixed‐frequency models and real‐time data to reassess the role of surveys and financial data relative to industrial production and orders in Germany. Although we find evidence that forecast characteristics based on real‐time and final data releases differ, we also observe minimal impacts on the relative forecasting performance of indicator models. However, when obtaining the optimal combination of soft and hard data, the use of final release data may understate the role of survey information.     

Forecasting South African inflation using non-linearmodels: a weighted loss-based evaluation

The conduct of inflation targeting is heavily dependent on accurate inflation forecasts. Non-linear models have increasingly featured, along with linear counterparts, in the forecasting literature. In this study, we focus on forecasting South African inflation by means of non-linear models and using a long historical dataset of seasonally adjusted monthly inflation rates spanning from 1921:02 to 2013:01. For an emerging market economy such as South Africa, non-linearities can be a salient feature of such long data, hence the relevance of evaluating non-linear models’ forecast performance. In the same vein, given the fact that 1969:10 marks the beginning of a protracted rising trend in South African inflation data, we estimate the models for an in-sample period of 1921:02–1966:09 and evaluate 1, 4, 12, and 24 step-ahead forecasts over an out-of-sample period of 1966:10–2013:01. In addition, using a weighted loss function specification, we evaluate the forecast performance of different non-linear models across various extreme economic environments and forecast horizons. In general, we find that no competing model consistently and significantly beats the LoLiMoT’s performance in forecasting South African inflation.     

Was the recent downturn in US real GDP predictable?

This article uses a small set of variables – real GDP, the inflation rate and the short-term interest rate – and a rich set of models – atheoretical (time series) and theoretical (structural), linear and nonlinear, as well as classical and Bayesian models – to consider whether we could have predicted the recent downturn of the US real GDP. Comparing the performance of the models to the benchmark random-walk model by root mean-square errors, the two structural (theoretical) models, especially the nonlinear model, perform well on average across all forecast horizons in our ex post, out-of-sample forecasts, although at specific forecast horizons certain nonlinear atheoretical models perform the best. The nonlinear theoretical model also dominates in our ex ante, out-of-sample forecast of the Great Recession, suggesting that developing forward-looking, microfounded, nonlinear, dynamic stochastic general equilibrium models of the economy may prove crucial in forecasting turning points.     

A boosting approach to forecasting gold and silver returns: economic and statistical forecast evaluation

We use a boosting algorithm to forecast the returns of gold and silver prices. We then study the implications of using different information criteria to terminate the boosting algorithm in terms of the statistical and economic performance of a forecasting model. Our findings demonstrate that information criteria that select parsimonious forecasting models perform better in statistical terms than information criteria that select relatively complex forecasting models, but this good performance does not necessarily survive an economic performance evaluation.