Static and dynamic models for multivariate distribution forecasts: Proper scoring rule tests of factor-quantile versus multivariate GARCH models

Many static and dynamic models exist to forecast Value-at-Risk and other quantile-related metrics used in financial risk management. Industry practice favours simpler, static models such as historical simulation or its variants. Most academic research focuses on dynamic models in the GARCH family. While numerous studies examine the accuracy of multivariate models for forecasting risk metrics, there is little research on accurately predicting the entire multivariate distribution. However, this is an essential element of asset pricing or portfolio optimization problems having non-analytic solutions. We approach this highly complex problem using various proper multivariate scoring rules to evaluate forecasts of eight-dimensional multivariate distributions: exchange rates, interest rates and commodity futures. This way, we test the performance of static models, namely, empirical distribution functions and a new factor-quantile model with commonly used dynamic models in the asymmetric multivariate GARCH class.     

Comparing the accuracy of multivariate density forecasts in selected regions of the copula support

This paper develops a testing framework for comparing the predictive accuracy of competing multivariate density forecasts with different predictive copulas, focusing on specific parts of the copula support. The tests are framed in the context of the Kullback–Leibler Information Criterion, using (out-of-sample) conditional likelihood and censored likelihood in order to focus the evaluation on the region of interest. Monte Carlo simulations document that the resulting test statistics have satisfactory size and power properties for realistic sample sizes. In an empirical application to daily changes of yields on government bonds of the G7 countries we obtain insights into why the Student-t and Clayton mixture copula outperforms the other copulas considered; mixing in the Clayton copula with the t-copula is of particular importance to obtain high forecast accuracy in periods of jointly falling yields.     

Evaluating density forecasts: forecast combinations,model mixtures,calibration and sharpness

This paper reviews current density forecast evaluation procedures, and considers a proposal that such procedures be augmented by an assessment of ‘sharpness’. This was motivated by an example in which some standard evaluation procedures using probability integral transforms cannot distinguish the ideal forecast from several competing forecasts. We show that this example has some unrealistic features from a time series forecasting perspective, and so provides insecure foundations for the argument that existing calibration procedures are inadequate in practice. Our alternative, more realistic example shows how relevant statistical methods, including information‐based methods, provide the required discrimination between competing forecasts. We introduce a new test of density forecast efficiency. Copyright © 2010 John Wiley & Sons, Ltd.     

Efficient forecast tests for conditional policy forecasts

Central Banks regularly make forecasts, such as the Fed’s Greenbook forecast, that are conditioned on hypothetical paths for the policy interest rate. While there are good public policy reasons to evaluate the quality of such forecasts, up until now, the most common approach has been to ignore their conditional nature and apply standard forecast efficiency tests. In this paper we derive tests for the efficiency of conditional forecasts. Intuitively, these tests involve implicit estimates of the degree to which the conditioning path is counterfactual and the magnitude of the policy feedback over the forecast horizon. We apply the tests to the Greenbook forecast and the Bank of England’s inflation report forecast, finding some evidence of forecast inefficiency. Nonetheless, we argue that the conditional nature of the forecasts made by central banks represents a substantial impediment to the analysis of their quality—stronger assumptions are needed and forecast inefficiency may go undetected for longer than would be the case if central banks were instead to report unconditional forecasts.     

Empirical tests of input-output forecasts

This paper analyzes previous studies of the accuracy of input-output forecasts as compared with projections derived from alternative forecasting techniques. The problem of constructing appropriate tests of input-output forecasts is discussed. Major tests of the interindustry approach and alternative techniques, such as final demand blowup, GNP blowup and multiple regression, conducted in the past four decades are reviewed and the major findings summarized. It is shown here that, contrary to the belief of some economists, the input-output forecasting model performs as well as and usually better than any of the alternatives considered.     

Kernel-based calibration diagnostics for recession and inflation probability forecasts

A probabilistic forecast is the estimated probability with which a future event will occur. One interesting feature of such forecasts is their calibration, or the match between the predicted probabilities and the actual outcome probabilities. Calibration has been evaluated in the past by grouping probability forecasts into discrete categories. We show here that we can do this without discrete groupings; the kernel estimators that we use produce efficiency gains and smooth estimated curves relating the predicted and actual probabilities. We use such estimates to evaluate the empirical evidence on the calibration error in a number of economic applications, including the prediction of recessions and inflation, using both forecasts made and stored in real time and pseudo-forecasts made using the data vintage available at the forecast date. The outcomes are evaluated using both first-release outcome measures and subsequent revised data. We find substantial evidence of incorrect calibration in professional forecasts of recessions and inflation from the SPF, as well as in real-time inflation forecasts from a variety of output gap models.     

Consistency of some tests for multivariate normality

Summary We prove that the tests of Cs?rgő (1986) and of Baringhaus and Henze (1988) for multivariate normality, both based on the empirical characteristic function, are consistent. Work partially supported by the Hungarian National Foundation for Scientific Research, Grants No. 1808/86 and 457/88.     

Forecast encompassing tests and probability forecasts

We consider tests of forecast encompassing for probability forecasts, for both quadratic and logarithmic scoring rules. We propose test statistics for the null of forecast encompassing, present the limiting distributions of the test statistics, and investigate the impact of estimating the forecasting models' parameters on these distributions. The small‐sample performance is investigated, in terms of small numbers of forecasts and model estimation sample sizes. We show the usefulness of the tests for the evaluation of recession probability forecasts from logit models with different leading indicators as explanatory variables, and for evaluating survey‐based probability forecasts. Copyright © 2009 John Wiley & Sons, Ltd.     

Likelihood-based scoring rules for comparing density forecasts in tails

We propose new scoring rules based on conditional and censored likelihood for assessing the predictive accuracy of competing density forecasts over a specific region of interest, such as the left tail in financial risk management. These scoring rules can be interpreted in terms of Kullback-Leibler divergence between weighted versions of the density forecast and the true density. Existing scoring rules based on weighted likelihood favor density forecasts with more probability mass in the given region, rendering predictive accuracy tests biased toward such densities. Using our novel likelihood-based scoring rules avoids this problem.     

Scoring rules and survey density forecasts

This article provides a practical evaluation of some leading density forecast scoring rules in the context of forecast surveys. We analyse the density forecasts of UK inflation obtained from the Bank of England’s Survey of External Forecasters, considering both the survey average forecasts published in the Bank’s quarterly Inflation Report, and the individual survey responses recently made available to researchers by the Bank. The density forecasts are collected in histogram format, and the ranked probability score (RPS) is shown to have clear advantages over other scoring rules. Missing observations are a feature of forecast surveys, and we introduce an adjustment to the RPS, based on the Yates decomposition, to improve its comparative measurement of forecaster performance in the face of differential non-response. The new measure, denoted RPS*, is recommended to analysts of forecast surveys.     

Combining density forecasts using focused scoring rules

We investigate the added value of combining density forecasts focused on a specific region of support. We develop forecast combination schemes that assign weights to individual predictive densities based on the censored likelihood scoring rule and the continuous ranked probability scoring rule (CRPS) and compare these to weighting schemes based on the log score and the equally weighted scheme. We apply this approach in the context of measuring downside risk in equity markets using recently developed volatility models, including HEAVY, realized GARCH and GAS models, applied to daily returns on the S&P 500, DJIA, FTSE and Nikkei indexes from 2000 until 2013. The results show that combined density forecasts based on optimizing the censored likelihood scoring rule significantly outperform pooling based on equal weights, optimizing the CRPS or log scoring rule. In addition, 99% Value‐at‐Risk estimates improve when weights are based on the censored likelihood scoring rule.     

A strategic predictive distribution for tests of probabilistic calibration

Forecasts of probability distributions are needed to support decision making in many applications. The accuracy of predictive distributions should be evaluated by maximising sharpness subject to calibration. Sharpness relates to the concentration of the predictive distributions, while calibration concerns their statistical consistency with the data. This paper focuses on calibration testing. It is important that a calibration test cannot be gamed by forecasts that have been strategically designed to pass the test. The widely used tests of probabilistic calibration for predictive distributions are based on the probability integral transform. Drawing on previous results for quantile prediction, we show that strategic distributional forecasting is a concern for these tests. To address this, we provide a simple extension of one of the tests. We illustrate ideas using simulated data.     

Comparing density forecasts in a risk management context

We compare multivariate and univariate approaches to assessing the accuracy of competing density forecasts of a portfolio return in the downside part of the support. We argue that the common practice of performing multivariate forecast comparisons can be problematic in the context of assessing portfolio risk, since better multivariate forecasts do not necessarily correspond to better aggregate portfolio return forecasts. This is illustrated by examples that involve (skew) elliptical distributions and an application to daily returns of a number of US stock prices. In addition, time-varying test statistics and Value-at-Risk forecasts provide empirical evidence of regime changes over the last decades.     

A Model Confidence Set approach to the combination of multivariate volatility forecasts

In predicting conditional covariance matrices of financial portfolios, practitioners are required to choose among several alternative options, facing a number of different sources of uncertainty. A first source is related to the frequency at which prices are observed, either daily or intradaily. Using prices sampled at higher frequency inevitably poses additional sources of uncertainty related to the selection of the optimal intradaily sampling frequency and to the construction of the best realized estimator. Likewise, the choices of model structure and estimation method also have a critical role. In order to alleviate the impact of these sources of uncertainty, we propose a forecast combination strategy based on the Model Confidence Set [MCS] to adaptively identify the set of most accurate predictors. The combined predictor is shown to achieve superior performance with respect to the whole model universe plus three additional competitors, independently of the MCS or portfolio settings.     

A comparison of financial duration models via density forecasts

Using density forecast evaluation techniques, we compare the predictive performance of econometric specifications that have been developed for modeling duration processes in intra-day financial markets. The model portfolio encompasses various variants of the Autoregressive Conditional Duration (ACD) model and recently proposed dynamic factor models. The evaluation is conducted on time series of trade, price and volume durations computed from transaction data of NYSE listed stocks. The results show that simpler approaches perform at least as well as more complex methods. With respect to modeling trade duration processes, standard ACD models successfully account for duration dynamics while none of the models provides an acceptable specification for the conditional duration distribution. We find that the Logarithmic ACD, if based on a flexible innovation distribution, provides a quite robust and useful framework for the modeling of price and volume duration processes.     

Evaluating the accuracy of value-at-risk forecasts: New multilevel tests

We propose independence and conditional coverage tests which are aimed at evaluating the accuracy of Value-at-Risk (VaR) forecasts from the same model at different confidence levels. The proposed procedures are multilevel tests, i.e., joint tests of several quantiles corresponding to different confidence levels. In a comprehensive Monte Carlo exercise, we document the superiority of the proposed tests with respect to existing multilevel tests. In an empirical application, we illustrate the implementation of the tests using several VaR models and daily data for 15 MSCI world indices.     

Generalized variance-ratio tests for serial correlation in multivariate regression models

Exact tests for rth order serial correlation in the multivariate linear regression model are devised which are based on a multivariate generalization of the F-distribution. The tests require the computation of two multivariate regressions. In the special case of a single-equation regression model the procedures reduce to simple always-conclusive F-tests. The tests are illustrated by applications to the Rotterdam Model of consumer demand.     

Robustifying multivariate trend tests to nonstationary volatility

This article studies inference of multivariate trend model when the volatility process is nonstationary. Within a quite general framework we analyze four classes of tests based on least squares estimation, one of which is robust to both weak serial correlation and nonstationary volatility. The existing multivariate trend tests, which either use non-robust standard errors or rely on non-standard distribution theory, are generally non-pivotal involving the unknown time-varying volatility function in the limit. Two-step residual-based i.i.d. bootstrap and wild bootstrap procedures are proposed for the robust tests and are shown to be asymptotically valid. Simulations demonstrate the effects of nonstationary volatility on the trend tests and the good behavior of the robust tests in finite samples.     

Distribution-free tests of mean vectors and covariance matrices for multivariate paired data

We study a permutation procedure to test the equality of mean vectors, homogeneity of covariance matrices, or simultaneous equality of both mean vectors and covariance matrices in multivariate paired data. We propose to use two test statistics for the equality of mean vectors and the homogeneity of covariance matrices, respectively, and combine them to test the simultaneous equality of both mean vectors and covariance matrices. Since the combined test has composite null hypothesis, we control its type I error probability and theoretically prove the asymptotic unbiasedness and consistency of the combined test. The new procedure requires no structural assumption on the covariances. No distributional assumption is imposed on the data, except that the permutation test for mean vector equality assumes symmetric joint distribution of the paired data. We illustrate the good performance of the proposed approach with comparison to competing methods via simulations. We apply the proposed method to testing the symmetry of tooth size in a dental study and to finding differentially expressed gene sets with dependent structures in a microarray study of prostate cancer.     

Local rank tests in a multivariate nonparametric relationship

Consider a multivariate nonparametric model where the unknown vector of functions depends on two sets of explanatory variables. For a fixed level of one set of explanatory variables, we provide consistent statistical tests, called local rank tests, to determine whether the multivariate relationship can be explained by a smaller number of functions. We also provide estimators for the smallest number of functions, called local rank, explaining the relationship. The local rank tests and the estimators of local rank are defined in terms of the eigenvalues of a kernel-based estimator of some matrix. The asymptotics of the eigenvalues is established by using the so-called Fujikoshi expansion along with some techniques of the theory of U-statistics. We present a simulation study which examines the small sample properties of local rank tests. We also apply the local rank tests and the local rank estimators to a demand system given by a newly constructed data set. This work can be viewed as a “local” extension of the tests for a number of factors in a nonparametric relationship introduced by Stephen Donald.