Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions

Verifying probabilistic forecasts for extreme events is a highly active research area because popular media and public opinions are naturally focused on extreme events, and biased conclusions are readily made. In this context, classical verification methods tailored for extreme events, such as thresholded and weighted scoring rules, have undesirable properties that cannot be mitigated, and the well-known continuous ranked probability score (CRPS) is no exception.In this paper, we define a formal framework for assessing the behavior of forecast evaluation procedures with respect to extreme events, which we use to demonstrate that assessment based on the expectation of a proper score is not suitable for extremes. Alternatively, we propose studying the properties of the CRPS as a random variable by using extreme value theory to address extreme event verification. An index is introduced to compare calibrated forecasts, which summarizes the ability of probabilistic forecasts for predicting extremes. The strengths and limitations of this method are discussed using both theoretical arguments and simulations.     

Comparing density forecasts in a risk management context

We compare multivariate and univariate approaches to assessing the accuracy of competing density forecasts of a portfolio return in the downside part of the support. We argue that the common practice of performing multivariate forecast comparisons can be problematic in the context of assessing portfolio risk, since better multivariate forecasts do not necessarily correspond to better aggregate portfolio return forecasts. This is illustrated by examples that involve (skew) elliptical distributions and an application to daily returns of a number of US stock prices. In addition, time-varying test statistics and Value-at-Risk forecasts provide empirical evidence of regime changes over the last decades.     

Evaluating factor forecasts for the UK: The role of asset prices

This paper applies a large data set, consisting of 167 monthly time series for the UK, both economic and financial, to simulate out-of-sample predictions of industrial production, inflation, 3-month Treasury Bills, and other variables. Fifteen dynamic factor models that allow forecasting based on large panels of time series are considered. The performances of these factor models are then compared to the following competing models: a simple univariate autoregressive, a vector autoregressive, a leading indicator, and a Phillips curve models. The results show that the best dynamic factor models outperform the competing models in forecasting at 6-, 12-, and 24-month horizons. Thus, the financial markets may have predictive power for the economic activity. This can be a useful tool for central banks and financial institutions, which may use the factor models to construct leading indicators of the economic conditions. In addition, researchers can see a strategic application of factor models.     

Testing the predictive accuracy of COVID-19 forecasts

We test the predictive accuracy of forecasts of the number of COVID-19 fatalities produced by several forecasting teams and collected by the United States Centers for Disease Control and Prevention for the epidemic in the United States. We find three main results. First, at the short horizon (1 week ahead) no forecasting team outperforms a simple time-series benchmark. Second, at longer horizons (3 and 4 week ahead) forecasters are more successful and sometimes outperform the benchmark. Third, one of the best performing forecasts is the Ensemble forecast, that combines all available predictions using uniform weights. In view of these results, collecting a wide range of forecasts and combining them in an ensemble forecast may be a superior approach for health authorities, rather than relying on a small number of forecasts.     

Statistical efficiency in the linear combination of forecasts

In seeking an efficient combination of forecasts which minimises the forecast error variance, many methods have been suggested. Through analysis, simulation and case studies, this paper seeks to develop insights into the statistical circumstances which influence the relative accuracy of six of these methods. The six methods chosen have all been advocated in various publications and consist of ‘equal weighting’ (i.e., pooled average), ‘optimal’ (i.e., error variance minimising), ‘optimal with independence assumption’ (i.e., error variance minimising assuming zero correlation between individual forecast errors) and three variations on the formulation of a Bayesian combination based upon posterior probabilities. The statistical circumstances reflected varying conditions of relative forecast errors, error correlations and outliers.     

A comparison of financial duration models via density forecasts

Using density forecast evaluation techniques, we compare the predictive performance of econometric specifications that have been developed for modeling duration processes in intra-day financial markets. The model portfolio encompasses various variants of the Autoregressive Conditional Duration (ACD) model and recently proposed dynamic factor models. The evaluation is conducted on time series of trade, price and volume durations computed from transaction data of NYSE listed stocks. The results show that simpler approaches perform at least as well as more complex methods. With respect to modeling trade duration processes, standard ACD models successfully account for duration dynamics while none of the models provides an acceptable specification for the conditional duration distribution. We find that the Logarithmic ACD, if based on a flexible innovation distribution, provides a quite robust and useful framework for the modeling of price and volume duration processes.     

Comparing the accuracy of multivariate density forecasts in selected regions of the copula support

This paper develops a testing framework for comparing the predictive accuracy of competing multivariate density forecasts with different predictive copulas, focusing on specific parts of the copula support. The tests are framed in the context of the Kullback–Leibler Information Criterion, using (out-of-sample) conditional likelihood and censored likelihood in order to focus the evaluation on the region of interest. Monte Carlo simulations document that the resulting test statistics have satisfactory size and power properties for realistic sample sizes. In an empirical application to daily changes of yields on government bonds of the G7 countries we obtain insights into why the Student-t and Clayton mixture copula outperforms the other copulas considered; mixing in the Clayton copula with the t-copula is of particular importance to obtain high forecast accuracy in periods of jointly falling yields.     

Combination of long term and short term forecasts, with application to tourism demand forecasting

Forecast combination is a well-established and well-tested approach for improving the forecasting accuracy. One beneficial strategy is to use constituent forecasts that have diverse information. In this paper we consider the idea of diversity being accomplished by using different time aggregations. For example, we could create a yearly time series from a monthly time series and produce forecasts for both, then combine the forecasts. These forecasts would each be tracking the dynamics of different time scales, and would therefore add diverse types of information. A comparison of several forecast combination methods, performed in the context of this setup, shows that this is indeed a beneficial strategy and generally provides a forecasting performance that is better than the performances of the individual forecasts that are combined.As a case study, we consider the problem of forecasting monthly tourism numbers for inbound tourism to Egypt. Specifically, we consider 33 individual source countries, as well as the aggregate. The novel combination strategy also produces a generally improved forecasting accuracy.     

Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters

This article provides a first analysis of the forecasts of inflation and GDP growth obtained from the Bank of England's Survey of External Forecasters, considering both the survey average forecasts published in the quarterly Inflation Report, and the individual survey responses, recently made available by the Bank. These comprise a conventional incomplete panel dataset, with an additional dimension arising from the collection of forecasts at several horizons; both point forecasts and density forecasts are collected. The inflation forecasts show good performance in tests of unbiasedness and efficiency, albeit over a relatively calm period for the UK economy, and there is considerable individual heterogeneity. For GDP growth, inaccurate real-time data and their subsequent revisions are seen to cause serious difficulties for forecast construction and evaluation, although the forecasts are again unbiased. There is evidence that some forecasters have asymmetric loss functions.     

Assessing the accuracy of electricity production forecasts in developing countries

This study assesses the accuracy of time series econometric methods for forecasting electricity production in developing countries. An analysis of the historical time series for 106 developing countries over the period 1960–2012 demonstrates that econometric forecasts are highly accurate for the majority of these countries. These forecasts have much smaller errors than the predictions of simple heuristic models, which assume that electricity production grows at an exogenous rate or is proportional to the real GDP growth. However, the quality of the forecasts diminishes for the countries and regions, where rapid economic and structural transformation makes it difficult to establish stable historical production trends.     

A copula-based time series model for global horizontal irradiation

The increasing importance of solar power for electricity generation leads to increasing demand for probabilistic forecasting of local and aggregated photovoltaic (PV) yields. Based on publicly available irradiation data, this paper uses an indirect modeling approach for hourly medium to long-term local PV yields. We suggest a time series model for global horizontal irradiation that allows for multivariate probabilistic forecasts for arbitrary time horizons. It features several important stylized facts. Sharp time-dependent lower and upper bounds of global horizontal irradiations are estimated. The parameters of the beta distributed marginals of the transformed data are allowed to be time-dependent. A copula-based time series model is introduced for the hourly and daily dependence structure based on simple vine copulas with so-called tail dependence. Evaluation methods based on scoring rules are used to compare the model’s power for multivariate probabilistic forecasting with other models used in the literature showing that our model outperforms other models in many respects.     

Verification results for age-structured models of economic–epidemics dynamics

In this paper we propose a macro-dynamic age-structured set-up for the analysis of epidemics/economic dynamics in continuous time.The resulting optimal control problem is reformulated in an infinite dimensional Hilbert space framework where we perform the basic steps of dynamic programming approach.Our main result is a verification theorem which allows to guess the feedback form of optimal strategies. This will be a departure point to discuss the behavior of the models of the family we introduce and their policy implications.     

Forecasting and loss functions

This paper considers two problems of interpreting forecasting competition error statistics. The first problem is concerned with the importance of linking the error measure (loss function) used in evaluating a forecasting model with the loss function used in estimating the model. It is argued that because the variety of uses of any single forecast, such matching is impractical. Secondly, there is little evidence that matching would have any impact on comparative forecast performance, however measured. As a consequence the results of forecasting competitions are not affected by this problem. The second problem is concerned with the interpreting performance, when evaluated through M(ean) S(quare) E(rror). The authors show that in the Makridakis Competition, good MSE performance is solely due to performance on a small number of the 1001 series, and arises because of the effects of scale. They conclude that comparisons of forecasting accuracy based on MSE are subject to major problems of interpretation.     

Nowcasting unemployment insurance claims in the time of COVID-19

Near-term forecasts, also called nowcasts, are most challenging but also most important when the economy experiences an abrupt change. In this paper, we explore the performance of models with different information sets and data structures in order to best nowcast US initial unemployment claims in spring of 2020 in the midst of the COVID-19 pandemic. We show that the best model, particularly near the structural break in claims, is a state-level panel model that includes dummy variables to capture the variation in timing of state-of-emergency declarations. Autoregressive models perform poorly at first but catch up relatively quickly. The state-level panel model, exploiting the variation in timing of state-of-emergency declarations, also performs better than models including Google Trends. Our results suggest that in times of structural change there is a bias–variance tradeoff. Early on, simple approaches to exploit relevant information in the cross sectional dimension improve forecasts, but in later periods the efficiency of autoregressive models dominates.     

Comparing the accuracy of several network-based COVID-19 prediction algorithms

Researchers from various scientific disciplines have attempted to forecast the spread of coronavirus disease 2019 (COVID-19). The proposed epidemic prediction methods range from basic curve fitting methods and traffic interaction models to machine-learning approaches. If we combine all these approaches, we obtain the Network Inference-based Prediction Algorithm (NIPA). In this paper, we analyse a diverse set of COVID-19 forecast algorithms, including several modifications of NIPA. Among the algorithms that we evaluated, the original NIPA performed best at forecasting the spread of COVID-19 in Hubei, China and in the Netherlands. In particular, we show that network-based forecasting is superior to any other forecasting algorithm.     

Reflections on Fourteen Cryptic Issues Concerning the Nature of Statistical Inference*

The present paper provides the original formulation and a joint response of a group of statistically trained scientists to fourteen cryptic issues for discussion, which were handed out to the public by Professor Dr. D.R. Cox after his Bernoulli Lecture 1997 at Groningen University.