Missing in Asynchronicity: A Kalman‐em Approach for Multivariate Realized Covariance Estimation

Motivated by the need for a positive‐semidefinite estimator of multivariate realized covariance matrices, we model noisy and asynchronous ultra‐high‐frequency asset prices in a state‐space framework with missing data. We then estimate the covariance matrix of the latent states through a Kalman smoother and expectation maximization (KEM) algorithm. Iterating between the two EM steps, we obtain a covariance matrix estimate which is robust to both asynchronicity and microstructure noise, and positive‐semidefinite by construction. We show the performance of the KEM estimator using extensive Monte Carlo simulations that mimic the liquidity and market microstructure characteristics of the S&P 500 universe as well as in a high‐dimensional application on US stocks. KEM provides very accurate covariance matrix estimates and significantly outperforms alternative approaches recently introduced in the literature. Copyright © 2014 John Wiley & Sons, Ltd.     

Efficiency of the California electricity reserves market

We test the efficiency of the California electricity reserves market by examining systematic differences between its day- and hour-ahead prices. We uncover significant day-ahead premia, which we attribute to market design characteristics. On the demand side, the market design established a principal–agent relationship between the markets' buyers (principal) and their supervisory authority (agent). The agent had very limited incentives to shift reserve purchases to the lower priced hour-ahead markets. On the supply side, the market design raised substantial entry barriers by precluding purely speculative trading and by introducing a complicated code of conduct that induced uncertainty about which actions were subject to regulatory scrutiny. We use a high-dimensional vector moving average model to estimate the premia and conduct correct inferences. To obtain exact maximum likelihood estimates of the model, we develop a new EM algorithm that seamlessly incorporates missing data and applies directly to general moving average time series models. Our algorithm uses only analytical expressions: the Kalman filter and a fixed interval smoother in the E step and least squares-type regressions in the M step. Copyright © 2007 John Wiley & Sons, Ltd.     

A look into the factor model black box: Publication lags and the role of hard and soft data in forecasting GDP

We derive forecast weights and uncertainty measures for assessing the roles of individual series in a dynamic factor model (DFM) for forecasting the euro area GDP from monthly indicators. The use of the Kalman smoother allows us to deal with publication lags when calculating the above measures. We find that surveys and financial data contain important information for the GDP forecasts beyond the monthly real activity measures. However, this is discovered only if their more timely publication is taken into account properly. Differences in publication lags play a very important role and should be considered in forecast evaluation.     

Do missing values exist? Incomplete data handling in cross-national longitudinal studies by means of continuous time modeling

In cross-national longitudinal studies it is often impossible to administer the same measurement instruments at the same occasions to all sample units in all participating countries. This quickly results in large quantities of missing data, due to (a) missing measurement instruments in some countries, (b) missing assessment waves within or across countries, (c) missing data for individual sample units. As compared to cross-sectional studies, the problem of missing values is further aggravated by the fact that missing values are always associated with different time intervals between repeated observations. In the past, this has often been dealt with by the use of phantom-variables, but this approach is limited to simple designs with few missing value patters. In the present paper we propose a new way to think of, and deal with, missing values in longitudinal studies. Instead of conceiving of a longitudinal study as a study with \(T\) discrete time points of which some are missing, we propose to conceive of a longitudinal study as a way to measure an underlying process that develops continuously over time, but is only observed at some selected discrete time points. This transforms the problem of missing values into a problem of unequal time intervals. After a quick introduction to the basic idea of continuous time modeling, we demonstrate how this approach provides a straightforward solution to missing measurement instruments in some countries, missing assessment waves within or across countries, and missing data for individual sample units.     

Assessing the relationships between Nationalism, Ethnocentrism, and Individualism in Flanders using Bergstrom's approximate discrete model

Although Nationalism, Ethnocentrism, and Individualism in Flanders have been the subject of several studies before, a longitudinal analysis has not been performed on all three concepts simultaneously nor have their relationships and the direction of their relationships been studied in continuous time. In this study we performed a continuous-time state-space analysis on panel data collected from 1274 subjects, in the years 1991, 1995 and 1999. The LISREL program is used for estimating the approximate discrete model (ADM), and for comparison, also the exact discrete model (EDM) is estimated by means of the Mx program. Details of continuous time modeling, especially the EDM and ADM, are dealt with. Individualism and Ethnocentrism turn out to be connected in a moderately strong feedback relationship with the effect from Individualism towards Ethnocentrism somewhat stronger than that in the opposite direction. Both Individualism and Ethnocentrism have small effects on Nationalism. The autoregression functions, cross-lagged effect functions, and mean predictions are shown.     

A two-step estimator for large approximate dynamic factor models based on Kalman filtering

This paper shows consistency of a two-step estimation of the factors in a dynamic approximate factor model when the panel of time series is large (n large). In the first step, the parameters of the model are estimated from an OLS on principal components. In the second step, the factors are estimated via the Kalman smoother. The analysis develops the theory for the estimator considered in Giannone et al. (2004) and Giannone et al. (2008) and for the many empirical papers using this framework for nowcasting.     

Maximum likelihood estimation for dynamic factor models with missing data

This paper concerns estimating parameters in a high-dimensional dynamic factor model by the method of maximum likelihood. To accommodate missing data in the analysis, we propose a new model representation for the dynamic factor model. It allows the Kalman filter and related smoothing methods to evaluate the likelihood function and to produce optimal factor estimates in a computationally efficient way when missing data is present. The implementation details of our methods for signal extraction and maximum likelihood estimation are discussed. The computational gains of the new devices are presented based on simulated data sets with varying numbers of missing entries.     

Identification with latent variables

This is an essay on a unified approach to the identifiability problem in static models with and without hidden endogenous variables. As is well known, when some of these variables are unobserved, the prior information requirements for models when all endogenous variables are observed, are still there. In addition, extra prior information that takes the place of the means and covariances of the missing variables will have to be supplied directly or indirectly by the statistical researcher. In the paper we characterize the quality and quantity of the required information for the general linear static model and apply it when the model is i) an econometric demand and supply model with missing observations on the quantity transacted, ii) a factor analysis model with observed characteristics of the test takers and iii) a LISREL Model without fixed exogenous variables. With unknown true parameters, the exact rank conditions are seldom verifiable but we do recommend an implementable check-list that is adequate for almost all parameters.     

Robust estimation of ARIMA models

The paper discusses methods of estimating univariate ARIMA models with outliers. The approach calls for a state vector representation of a time-series model, on which we can then operate on using the Kalman filter. One of the additional advantages of Kalman filter operating on the state vector representation is that the method and code could easily be adapted to be applicable to the ARIMA model with missing observations. The paper investigates ways to calculate robust initial estimation of the parameters of the ARIMA model. The method proposed is based on the results obtained by R.D. Martin (1980).     

Optimal combination of Arctic sea ice extent measures: A dynamic factor modeling approach

The diminishing extent of Arctic sea ice is a key indicator of climate change as well as being an accelerant for future global warming. Since 1978, Arctic sea ice has been measured using satellite-based microwave sensing; however, different measures of Arctic sea ice extent have been made available based on differing algorithmic transformations of raw satellite data. We propose and estimate a dynamic factor model that combines four of these measures in an optimal way and accounts for their differing volatility and cross-correlations. We then use the Kalman smoother to extract an optimal combined measure of Arctic sea ice extent. It turns out that almost all weight is put on the NSIDC Sea Ice Index, confirming and enhancing confidence in the Sea Ice Index and the NASA Team algorithm on which it is based.     

Discovering Statistically Significant Periodic Gene Expression

One frequent application of microarray experiments is in the study of monitoring gene activities in a cell during cell cycle or cell division. High throughput gene expression time series data are produced from such microarray experiments. A new computational and statistical challenge for analyzing such gene expression time course data, resulting from cell cycle microarray experiments, is to discover genes that are statistically significantly periodically expressed during the cell cycle. Such a challenge occurs due to the large number of genes that are simultaneously measured, a moderate to small number of measurements per gene taken at different time points and high levels of non-normal random noises inherited in the data. Computational and statistical approaches to discovery and validation of periodic patterns of gene expression are, however, very limited. A good method of analysis should be able to search for significant periodic genes with a controlled family-wise error (FWE) rate or controlled false discovery rate (FDR) and any other variations of FDR, when all gene expression profiles are compared simultaneously. In this review paper, a brief summary of currently used methods in searching for periodic genes will be given. In particular, two methods will be surveyed in details. The first one is a novel statistical inference approach, the C & G Procedure that can be used to effectively detect statistically significantly periodically expressed genes when the gene expression is measured on evenly spaced time points. The second one is the Lomb–Scargle periodogram analysis, which can be used to discover periodic genes when the gene profiles are not measured on evenly spaced time points or when there are missing values in the profiles. The ultimate goal of this review paper is to give an expository of the two surveyed methods to researchers in related fields.     

Methods for the Analysis of Explanatory Linear Regression Models with Missing Data Not at Random

Since the work of Little and Rubin (1987) not substantial advances in the analysisof explanatory regression models for incomplete data with missing not at randomhave been achieved, mainly due to the difficulty of verifying the randomness ofthe unknown data. In practice, the analysis of nonrandom missing data is donewith techniques designed for datasets with random or completely random missingdata, as complete case analysis, mean imputation, regression imputation, maximumlikelihood or multiple imputation. However, the data conditions required to minimizethe bias derived from an incorrect analysis have not been fully determined. In thepresent work, several Monte Carlo simulations have been carried out to establishthe best strategy of analysis for random missing data applicable in datasets withnonrandom missing data. The factors involved in simulations are sample size,percentage of missing data, predictive power of the imputation model and existenceof interaction between predictors. The results show that the smallest bias is obtainedwith maximum likelihood and multiple imputation techniques, although with lowpercentages of missing data, absence of interaction and high predictive power ofthe imputation model (frequent data structures in research on child and adolescentpsychopathology) acceptable results are obtained with the simplest regression imputation.     

虚拟人力资源管理与企业绩效关系的实证研究

本文对虚拟人力资源管理的概念、类型进行了界定,以陕西省135家企业为研究对象,利用SPSS和LISREL软件,基于实证数据调研,探讨虚拟人力资源管理的两种虚拟类型对企业绩效是否有正向影响。研究结果表明:不同虚拟类型对陕西企业绩效均有一定正向影响,且技术虚拟影响程度大于组织虚拟。     

Modelling the spatiotemporal distribution of the incidence of resident foreign population

Many European countries have recently experienced a substantial increase in the proportion of immigrants in their populations. The incidence of resident foreigners calculated at a national level does not provide information on the local spatial and temporal distribution of the phenomenon. This information may be of crucial importance for planning local policies. In this article, we suggest a tool for practitioners to provide spatiotemporal maps representing the local distribution of the incidence of resident foreigners in the territory, and changes in spatial trends over time. We illustrate this with Italian data at a municipal level, for the period 2003–2008. To account for spatiotemporal interactions in the data, we propose using a generalized additive model incorporating a smoother of the time and space dimensions. Specifically, we set up a tensor product smoother combining a cubic regression spline basis for time and a soap film spline basis for space. This approach provides a consistent framework to produce spatiotemporal maps which could be effectively used by policy makers to decide the allocation of economic resources at a local level.     

Factor extraction using Kalman filter and smoothing: This is not just another survey

Dynamic factor models have been the main “big data” tool used by empirical macroeconomists during the last 30 years. In this context, Kalman filter and smoothing (KFS) procedures can cope with missing data, mixed frequency data, time-varying parameters, non-linearities, non-stationarity, and many other characteristics often observed in real systems of economic variables. The main contribution of this paper is to provide a comprehensive updated summary of the literature on latent common factors extracted using KFS procedures in the context of dynamic factor models, pointing out their potential limitations. Signal extraction and parameter estimation issues are separately analyzed. Identification issues are also tackled in both stationary and non-stationary models. Finally, empirical applications are surveyed in both cases. This survey is relevant to researchers and practitioners interested not only in the theory of KFS procedures for factor extraction in dynamic factor models but also in their empirical application in macroeconomics and finance.     

Handling Missing Data by Re-approaching Non-respondents

When handling missing data, a researcher should be aware of the mechanism underlying the missingness. In the presence of non-randomly missing data, a model of the missing data mechanism should be included in the analyses to prevent the analyses based on the data from becoming biased. Modeling the missing data mechanism, however, is a difficult task. One way in which knowledge about the missing data mechanism may be obtained is by collecting additional data from non-respondents. In this paper the method of re-approaching respondents who did not answer all questions of a questionnaire is described. New answers were obtained from a sample of these non-respondents and the reason(s) for skipping questions was (were) probed for. The additional data resulted in a larger sample and was used to investigate the differences between respondents and non-respondents, whereas probing for the causes of missingness resulted in more knowledge about the nature of the missing data patterns.     

Alternative measures of political efficacy: models and means

In the 1982 and 1984 American national election surveys the CPS deleted a subset of its longstanding measures of political efficacy. This paper employs covariance structure analysis to test a two-factor measurement model based on alternative indicators. The model has an excellent fit for data from the 1964–84 national surveys as well as for different educational, gender, and racial groups in the 1984 study. Consistent with previous theorizing, the internal efficacy factor is significantly more stable than the external factor. Also, the two efficacy factors correlate as expected with measures of general personal competence and political trust. The paper is the first to use LISREL structured means tests to test group differences in levels of efficacy or other important political attitudes. Since this technique is not well-known, a tutorial appendix describing its implementation in LISREL VI is included to facilitate future research.     

Missing observations in ARIMA models: Skipping approach versus additive outlier approach

Optimal estimation of missing values in ARMA models is typically performed by using the Kalman filter for likelihood evaluation, ‘skipping’ in the computations the missing observations, obtaining the maximum likelihood (ML) estimators of the model parameters, and using some smoothing algorithm. The same type of procedure has been extended to nonstationary ARIMA models in Gómez and Maravall (1994). An alternative procedure suggests filling in the holes in the series with arbitrary values and then performing ML estimation of the ARIMA model with additive outliers (AO). When the model parameters are not known the two methods differ, since the AO likelihood is affected by the arbitrary values. We develop the proper likelihood for the AO approach in the general non-stationary case and show the equivalence of this and the skipping method. Finally, the two methods are compared through simulation, and their relative advantages assessed; the comparison also includes the AO method with the uncorrected likelihood.     

Sequential control of time series by functionals of kernal-weighted empirical processes under local alternatives

Motivated in part by applications in model selection in statistical genetics and sequential monitoring of financial data, we study an empirical process framework for a class of stopping rules which rely on kernel-weighted averages of past data. We are interested in the asymptotic distribution for time series data and an analysis of the joint influence of the smoothing policy and the alternative defining the deviation from the null model (in-control state). We employ a certain type of local alternative which provides meaningful insights. Our results hold true for short memory processes which satisfy a weak mixing condition. By relying on an empirical process framework we obtain both asymptotic laws for the classical fixed sample design and the sequential monitoring design. As a by-product we establish the asymptotic distribution of the Nadaraya-Watson kernel smoother when the regressors do not get dense as the sample size increases.Acknowledgements The author is grateful to two anonymous referees for their constructive comments, which improved the paper. One referee draws my attention to Lifshits paper. The financial support of the Collaborative Research Centre Reduction of Complexity in Multivariate Data Structures (SFB 475) of the German Research Foundation (DFG) is greatly acknowledged.     

Unemployment or credit: Which one holds the potential? Results for a small open economy with a low degree of financialization

This paper investigates the relative importance of unemployment versus credit in determining the potential level of real activity for a small open economy with a low degree of financialization. We use a multivariate unobserved component model (MUC) to derive the potential output and the associated output gap for the Lithuanian economy. The model is estimated via Bayesian methods and the time paths of unobserved variables are extracted via the Kalman filter. The inclusion of unemployment in the MUC model substantially improves the estimates of the output gap in real time. Adding information about credit further emphasizes the overheating of the economy in the pre-crisis period, both in real time and ex post. Including credit preserves the conclusions regarding turning points. We uncover a strong negative correlation between the model-implied unemployment gap (without accounting for credit) and real credit growth. Data revisions do not appear to be the primary source of revisions of output gap estimates.