Forecast Encompassing and Parameter Estimation*

A desirable property of a forecast is that it encompasses competing predictions, in the sense that the accuracy of the preferred forecast cannot be improved through linear combination with a rival prediction. In this paper, we investigate the impact of the uncertainty associated with estimating model parameters in‐sample on the encompassing properties of out‐of‐sample forecasts. Specifically, using examples of non‐nested econometric models, we show that forecasts from the true (but estimated) data generating process (DGP) do not encompass forecasts from competing mis‐specified models in general, particularly when the number of in‐sample observations is small. Following this result, we also examine the scope for achieving gains in accuracy by combining the forecasts from the DGP and mis‐specified models.     

Forecasting combination and encompassing tests

In this paper we demonstrate that forecast encompassing tests are valuable tools in getting an insight into why competing forecasts may be combined to produce a composite forecast which is superior to the individual forecasts. We also argue that results from forecast encompassing tests are potentially useful in model specification. We illustrate this using forecasts of quarterly UK consumption expenditure data from three classes of models: ARIMA, DHSY and VAR models.     

Testing forecast accuracy of expectiles and quantiles with the extremal consistent loss functions

Forecast evaluations aim to choose an accurate forecast for making decisions by using loss functions. However, different loss functions often generate different ranking results for forecasts, which complicates the task of comparisons. In this paper, we develop statistical tests for comparing performances of forecasting expectiles and quantiles of a random variable under consistent loss functions. The test statistics are constructed with the extremal consistent loss functions of Ehm et al. (2016). The null hypothesis of the tests is that a benchmark forecast at least performs equally well as a competing one under all extremal consistent loss functions. It can be shown that if such a null holds, the benchmark will also perform at least equally well as the competitor under all consistent loss functions. Thus under the null, when different consistent loss functions are used, the result that the competitor does not outperform the benchmark will not be altered. We establish asymptotic properties of the proposed test statistics and propose to use the re-centered bootstrap to construct their empirical distributions. Through simulations, we show that the proposed test statistics perform reasonably well. We then apply the proposed method to evaluations of several different forecast methods.     

Tests for multiple forecast encompassing

In the evaluation of economic forecasts, it is frequently the case that comparisons are made between a number of competing predictors. A natural question to ask in such contexts is whether one forecast encompasses its competitors, in the sense that they contain no useful information not present in the superior forecast. We develop tests for this notion of multiple forecast encompassing which are robust to properties expected in the forecast errors, and apply the tests to forecasts of UK growth and inflation. Copyright © 2000 John Wiley & Sons, Ltd.     

Forecast encompassing tests for the expected shortfall

We introduce new forecast encompassing tests for the risk measure Expected Shortfall (ES). The ES has received much attention since its introduction into the Basel III Accords, which stipulate its use as the primary market risk measure for international banking regulation. We utilize joint loss functions for the pair ES and Value at Risk to set up three ES encompassing test variants. The tests are built on an asymptotic theory that is robust to misspecifications. We investigate the finite sample properties of the tests in an extensive simulation study. Finally, we use the encompassing tests to illustrate the potential of forecast combination methods for different financial assets.     

Likelihood‐Based Inference and Prediction in Spatio‐Temporal Panel Count Models for Urban Crimes

We develop a panel count model with a latent spatio‐temporal heterogeneous state process for monthly severe crimes at the census‐tract level in Pittsburgh, Pennsylvania. Our dataset combines Uniform Crime Reporting data with socio‐economic data. The likelihood is estimated by efficient importance sampling techniques for high‐dimensional spatial models. Estimation results confirm the broken‐windows hypothesis whereby less severe crimes are leading indicators for severe crimes. In addition to ML parameter estimates, we compute several other statistics of interest for law enforcement such as spatio‐temporal elasticities of severe crimes with respect to less severe crimes, out‐of‐sample forecasts, predictive distributions and validation test statistics. Copyright © 2016 John Wiley & Sons, Ltd.     

Comparing density forecasts in a risk management context

We compare multivariate and univariate approaches to assessing the accuracy of competing density forecasts of a portfolio return in the downside part of the support. We argue that the common practice of performing multivariate forecast comparisons can be problematic in the context of assessing portfolio risk, since better multivariate forecasts do not necessarily correspond to better aggregate portfolio return forecasts. This is illustrated by examples that involve (skew) elliptical distributions and an application to daily returns of a number of US stock prices. In addition, time-varying test statistics and Value-at-Risk forecasts provide empirical evidence of regime changes over the last decades.     

Combining probability forecasts

We consider different methods for combining probability forecasts. In empirical exercises, the data generating process of the forecasts and the event being forecast is not known, and therefore the optimal form of combination will also be unknown. We consider the properties of various combination schemes for a number of plausible data generating processes, and indicate which types of combinations are likely to be useful. We also show that whether forecast encompassing is found to hold between two rival sets of forecasts or not may depend on the type of combination adopted. The relative performances of the different combination methods are illustrated, with an application to predicting recession probabilities using leading indicators.     

Jumps in cross‐sectional rank and expected returns: a mixture model

We propose a new nonlinear time series model of expected returns based on the dynamics of the cross‐sectional rank of realized returns. We model the joint dynamics of a sharp jump in the cross‐sectional rank and the asset return by analyzing (1) the marginal probability distribution of a jump in the cross‐sectional rank within the context of a duration model, and (2) the probability distribution of the asset return conditional on a jump, for which we specify different dynamics depending upon whether or not a jump has taken place. As a result, the expected returns are generated by a mixture of normal distributions weighted by the probability of jumping. The model is estimated for the weekly returns of the constituents of the SP500 index from 1990 to 2000, and its performance is assessed in an out‐of‐sample exercise from 2001 to 2005. Based on the one‐step‐ahead forecast of the mixture model we propose a trading rule, which is evaluated according to several forecast evaluation criteria and compared to 18 alternative trading rules. We find that the proposed trading strategy is the dominant rule by providing superior risk‐adjusted mean trading returns and accurate value‐at‐risk forecasts. Copyright © 2008 John Wiley & Sons, Ltd.     

Does Forecast Combination Improve Norges Bank Inflation Forecasts?*

We develop a system that provides model‐based forecasts for inflation in Norway. We recursively evaluate quasi out‐of‐sample forecasts from a large suite of models from 1999 to 2009. The performance of the models are then used to derive quasi real time weights that are used to combine the forecasts. Our results indicate that a combination forecast improves upon the point forecasts from individual models. Furthermore, a combination forecast outperforms Norges Bank's own point forecast for inflation. The beneficial results are obtained using a trimmed weighted average. Some degree of trimming is required for the combination forecasts to outperform the judgmental forecasts from the policymaker.     

The usefulness of oil price forecasts—Evidence from survey predictions

This paper evaluates survey forecasts for crude oil prices and discusses the implications for decision makers. A novel disaggregated data set incorporating individual forecasts for Brent and Western Texas Intermediate is used. We carry out tests for unbiasedness, sign accuracy, and forecast encompassing, followed by the computation of coefficients for topically oriented trend adjustments and the Theil's U measure. We also control for the forecast horizon finding heterogeneous results. Forecasts are more precise for shorter horizons, but less accurate than the naïve prediction. For longer horizons, topically oriented trend adjustments become more pronounced, but forecasters tend to outperform the naïve predictions.     

Composite forecasts of inflation: An improvement in forecasting performance

This study investigates the performance of a composite forecast of inflation for the period 1969:I–1992:IV. This composite forecast is generated by combining the forecasts of four methods commonly used to measure expected inflation. Initially, the results of conditional efficiency tests suggest that a composite forecast can improve performance by encompassing a wider information set. The evidence, from the comparison of various forecast series, shows that the composite forecast improves on the performance of the four individual forecasts and an alternative composite forecast in terms of accuracy and rationality criteria.     

Evaluating density forecasts: forecast combinations,model mixtures,calibration and sharpness

This paper reviews current density forecast evaluation procedures, and considers a proposal that such procedures be augmented by an assessment of ‘sharpness’. This was motivated by an example in which some standard evaluation procedures using probability integral transforms cannot distinguish the ideal forecast from several competing forecasts. We show that this example has some unrealistic features from a time series forecasting perspective, and so provides insecure foundations for the argument that existing calibration procedures are inadequate in practice. Our alternative, more realistic example shows how relevant statistical methods, including information‐based methods, provide the required discrimination between competing forecasts. We introduce a new test of density forecast efficiency. Copyright © 2010 John Wiley & Sons, Ltd.     

CONSTRUCTING OPTIMAL DENSITY FORECASTS FROM POINT FORECAST COMBINATIONS

Decision makers often observe point forecasts of the same variable computed, for instance, by commercial banks, IMF and the World Bank, but the econometric models used by such institutions are frequently unknown. This paper shows how to use the information available on point forecasts to compute optimal density forecasts. Our idea builds upon the combination of point forecasts under general loss functions and unknown forecast error distributions. We use real‐time data to forecast the density of US inflation. The results indicate that the proposed method materially improves the real‐time accuracy of density forecasts vis‐à‐vis those from the (unknown) individual econometric models. Copyright © 2013 John Wiley & Sons, Ltd.     

A strategic predictive distribution for tests of probabilistic calibration

Forecasts of probability distributions are needed to support decision making in many applications. The accuracy of predictive distributions should be evaluated by maximising sharpness subject to calibration. Sharpness relates to the concentration of the predictive distributions, while calibration concerns their statistical consistency with the data. This paper focuses on calibration testing. It is important that a calibration test cannot be gamed by forecasts that have been strategically designed to pass the test. The widely used tests of probabilistic calibration for predictive distributions are based on the probability integral transform. Drawing on previous results for quantile prediction, we show that strategic distributional forecasting is a concern for these tests. To address this, we provide a simple extension of one of the tests. We illustrate ideas using simulated data.     

Forecast Rationality Tests in the Presence of Instabilities,with Applications to Federal Reserve and Survey Forecasts

This paper proposes a framework to implement regression‐based tests of predictive ability in unstable environments, including, in particular, forecast unbiasedness and efficiency tests, commonly referred to as tests of forecast rationality. Our framework is general: it can be applied to model‐based forecasts obtained either with recursive or rolling window estimation schemes, as well as to forecasts that are model free. The proposed tests provide more evidence against forecast rationality than previously found in the Federal Reserve's Greenbook forecasts as well as survey‐based private forecasts. It confirms, however, that the Federal Reserve has additional information about current and future states of the economy relative to market participants. Copyright © 2015 John Wiley & Sons, Ltd.     

Testing cointegration relationship in a semiparametric varying coefficient model

In this paper, we develop two cointegration tests for two varying coefficient cointegration regression models, respectively. Our test statistics are residual based. We derive the asymptotic distributions of test statistics under the null hypothesis of cointegration and show that they are consistent against the alternative hypotheses. We also propose a wild bootstrap procedure companioned with the continuous moving block bootstrap method proposed in  Paparoditis and Politis (2001) and  Phillips (2010) to rectify severe distortions found in simulations when the sample size is small. We apply the proposed test statistic to examine the purchasing power parity (PPP) hypothesis between the US and Canada. In contrast to the existing results from linear cointegration tests, our varying coefficient cointegration test does not reject that PPP holds between the US and Canada.     

Forecasting dynamic return distributions based on ordered binary choice

We present a simple approach to the forecasting of conditional probability distributions of asset returns. We work with a parsimonious specification of ordered binary choice regressions that imposes a connection on sign predictability across different quantiles. The model forecasts the future conditional probability distributions of returns quite precisely when using a past indicator and a past volatility proxy as predictors. The direct benefits of the model are revealed in an empirical application to the 29 most liquid U.S. stocks. The forecast probability distribution is translated to significant economic gains in a simple trading strategy. Our approach can also be useful in many other applications in which conditional distribution forecasts are desired.     

Managerial evaluation of exogenous forecast sources

This paper concerns the managerial evaluation of forecast vendors—individuals or firms offering for sale future forecasts of random variables relevant to managerial decision making. Assuming the forecasts are exogenous in the sense they are generated by a methodology unknown or unproven to management, the paper uses a logistic regression model to present a statistical test for informativeness that allows for an interpretation of the vendor's abilities. The advantage of the approach is that it requires as input only knowledge of the unconditional probability distribution of the variable being forecast and a relatively small historical track record of the vendor's forecasting performance. No benchmark forecast is necessary and few assumptions are required about the statistical process that generates the forecasts. As an illustrative empirical application, the paper presents an evaluation of the informativeness of the published long-range price forecasts by a veteran analyst of the Iowa hog market.     

ON COMPARING MACROECONOMIC MODELS USING FORECAST ENCOMPASSING TESTS

ABSTRACT It is clearly of interest to macroeconomists to be able to evaluate whether one large-scale macroeconometric model ‘is better’ than another. Although comparisons between models are sometimes invidious, because the purposes for which the models were built differ, it is the case that formal comparisons of two models' statistical properties are rare. This is in spite of considerable theoretical advances in the econometric methodology, namely the development and use of non-nested and encompassing tests. Chong and Hendry (1986) advocate the use of the forecast encompassing regressions, where the outturns are regressed on competing (one-step-ahead) forecasts. This paper reports the findings of applying this rather easy-to-use method of comparing large scale macroeconometric models. The forecast data we use are those published by three macroeconometric modelling groups, namely: Liverpool; the National Institute; and The London Business School. Forecasts for up to three years ahead are published for unemployment, growth, and inflation, throughout the 1980's. Forecast encompassing tests fail to separate one model from another, based on one-year-ahead forecasts. Each model ‘wins’ once. However, the conclusions are not the same as using root-mean-square-forecast-error criteria, illustrating Clements and Hendry's (1994) observation that minimum root-mean-square-forecast-error is neither necessary nor sufficient for a model to have constant parameters, to provide accurate forecasts, or to encompass its rivals.