An overview of variance component estimation

Variance components estimation originated with estimating error variance in analysis of variance by equating error mean square to its expected value. This equating procedure was then extended to random effects models, first for balanced data (for which minimum variance properties were subsequently established) and later for unbalanced data. Unfortunately, this ANOVA methodology yields no optimum properties (other than unbiasedness) for estimation from unbalanced data. Today it is being replaced by maximum likelihood (ML) and restricted maximum likelihood (REML) based on normality assumptions and involving nonlinear equations that have to be solved numerically. There is also minimum norm quadratic unbiased estimation (MINQUE) which is closely related to REML but with fewer advantages.An invited paper for the ProbaStat '94 conference, Smolenice, Slovakia, May 30–June 3, 1994 Paper number BU-677 in the Biometrics Unit. Cornell University Ithaca NY     

Analysis of a mixed model for ordinal data by iterative re-weighted REML

An estimation procedure will be presented for a class of threshold models for ordinal data. These models may include both fixed and random effects with associated components of variance on an underlying scale. The residual error distribution on the underlying scale may be rendered greater flexibility by introducing additional shape parameters, e.g. a kurtosis parameter or parameters to model heterogeneous residual variances as a function of factors and covariates. The estimation procedure is an extension of an iterative re-weighted restricted maximum likelihood procedure, originally developed for generalized linear mixed models. This procedure will be illustrated with a practical problem involving damage to potato tubers and with data from animal breeding and medical research from the literature.     

A distribution-free least squares estimator for censored linear regression models

This paper describes a method for estimating simultaneously the parameter vector of the systematic component and the distribution function of the random component of a censored linear regression model. The estimator is obtained by minimizing the sum of the squares of the differences between the observed values of the dependent variable and the corresponding expected values of this variable according to the estimated parameter vector and distribution function. The resulting least squares parameter estimator incorporates information on the distribution of the random component of the regression model that is available from the estimation sample. Hence, it may often be more efficient than are parameter estimators that do not use such information. The results of numerical experiments with the least squares estimator tend to support this hypothesis.     

Prediction of the next hypercalcemia free period: application of random effect models with selection on first event

The length of repeated hypercalcemia free periods of patients with bone metastasis of breast cancer with at least one hypercalcemic event was modelled according to a generalized linear mixed model formulated in terms of transition probabilities and according to a latent variable model. In the former case the periods were assumed to be lognormally distributed with two variance components (patients and residue). In the latter case the conditional intensity given a patient was assumed to be the intensity of the Weibull distribution, while the random patient effect (frailty) was assumed to be drawn from a gamma distribution. In both cases the selection of only patients with at least one hypercalcemic event was taken into consideration. In both models the variance of the patient effect turned out to be negligible. For the second and later periods the Weibull appeared to fit better than the lognormal model. For the first period there was almost no information available.     

Improved forecasting of autoregressive series by weighted least squares approximate REML estimation

Restricted maximum likelihood (REML) estimation has recently been shown to provide less biased estimates in autoregressive series. A simple weighted least squares approximate REML procedure has been developed that is particularly useful for vector autoregressive processes. Here, we compare the forecasts of such processes using both the standard ordinary least squares (OLS) estimates and the new approximate REML estimates. Forecasts based on the approximate REML estimates are found to provide a significant improvement over those obtained using the standard OLS estimates.     

Fixed Effects and Random Effects Estimation of Higher-order Spatial Autoregressive Models with Spatial Autoregressive and Heteroscedastic Disturbances

Abstract

This paper develops a unified framework for fixed effects (FE) and random effects (RE) estimation of higher-order spatial autoregressive panel data models with spatial autoregressive disturbances and heteroscedasticity of unknown form in the idiosyncratic error component. We derive the moment conditions and optimal weighting matrix without distributional assumptions for a generalized moments (GM) estimation procedure of the spatial autoregressive parameters of the disturbance process and define both an RE and an FE spatial generalized two-stage least squares estimator for the regression parameters of the model. We prove consistency of the proposed estimators and derive their joint asymptotic distribution, which is robust to heteroscedasticity of unknown form in the idiosyncratic error component. Finally, we derive a robust Hausman test of the spatial random against the spatial FE model.     

Quasi-likelihood inference in a generalized linear mixed model for balanced data

For modelling the effect of crossed, fixed factors on the response variable in balanced designs with nested stratifications, a generalized linear mixed model is proposed. This model is based on a set of quasi-likelihood assumptions which imply quadratic variance functions. From these variance functions, deviances are obtained to quantify the variation per stratification. The effects of the fixed factors will be tested, an dispersion components will be estimated. The practical use of the model is illustrated by reanalysing a soldering failures problem.     

Time-series–cross-section Data

This article treats the analysis of 'time-series–cross-section' (TSCS) data. Such data consists of repeated observations on a series of fixed units. Examples of such data are annual observations on the political economy of OECD nations in the post-war era. TSCS data is distinguished from 'panel' data, in that asymptotics are in the number of repeated observations, not the number of units.
The article begins by treating the complications of TSCS data in an 'old-fashioned' manner, that is, as a nuisance which causes estimation difficulties. It claims that TSCS data should be analyzed via ordinary least squares with 'panel correct standard errors' rather than generalized least squares methods. Dynamics should be modeled via a lagged dependent variable or, if appropriate, a single equation error correction model.
The article then treats more modern issues, in particular, the modeling of spatial effects and heterogeneity. It also claims that heterogeneity should be assessed with 'panel cross-validation' as well as more standard tests. The article concludes with a discussion of estimation in the presence of a binary dependent variable.     

On the performance of the ordinary least squares method under an error component model

Starting from the one-dimensional results by Wang et al (1994) we consider the performance of the ordinary least squares estimator in comparison to the best linear unbiased estimator under an error component model with random effects in units and time. Upper bounds are derived for the first-order approximation to the difference between both estimators and for the spectral norm of the difference between their dispersion matrices.     

Analysis of Categorical Data for Complex Surveys

This paper reviews methods for handling complex sampling schemes when analysing categorical survey data. It is generally assumed that the complex sampling scheme does not affect the specification of the parameters of interest, only the methodology for making inference about these parameters. The organisation of the paper is loosely chronological. Contingency table data are emphasised first before moving on to the analysis of unit‐level data. Weighted least squares methods, introduced in the mid 1970s along with methods for two‐way tables, receive early attention. They are followed by more general methods based on maximum likelihood, particularly pseudo maximum likelihood estimation. Point estimation methods typically involve the use of survey weights in some way. Variance estimation methods are described in broad terms. There is a particular emphasis on methods of testing. The main modelling methods considered are log‐linear models, logit models, generalised linear models and latent variable models. There is no coverage of multilevel models.     

Tests for model specification in the presence of alternative hypotheses: Some further results

In Davidson and MacKinnon (1981), two of the present authors proposed a novel and very simple procedure for testing the specification of a nonlinear regression model against the evidence provided by a non-nested alternative. In this paper we extend their results in several directions. First, we relax a number of the assumptions of the previous paper; we admit the possibility that the nonlinear regression functions may depend on lagged dependent variables, and we do not require that the error terms be normally distributed. Second, we show how the earlier procedure may straightforwardly be generalized to the case where the two non-nested models involve different transformations of the dependent variable. Finally, we propose a simple procedure for testing non-nested linear regression models which have endogenous variables on the right-hand side, and have therefore been estimated by two-stage least squares.     

Restricted maximum likelihood estimation under Eisenhart model Ill

For a balanced two-way mixed model, the maximum likelihood (ML) and restricted ML (REML) estimators of the variance components were obtained and compared under the non-negativity requirements of the variance components by L ee and K apadia (1984). In this note, for a mixed (random blocks) incomplete block model, explicit forms for the REML estimators of variance components are obtained. They are always non-negative and have smaller mean squared error (MSE) than the analysis of variance (AOV) estimators. The asymptotic sampling variances of the maximum likelihood (ML) estimators and the REML estimators are compared and the balanced incomplete block design (BIBD) is considered as a special case. The ML estimators are shown to have smaller asymptotic variances than the REML estimators, but a numerical result in the randomized complete block design (RCBD) demonstrated that the performances of the REML and ML estimators are not much different in the MSE sense.     

An unexpected property of minimax estimation in the relative squared error approach to linear regression analysis

An unexpected property of the relative squared error approach to linear regression analysis is derived: It is shown that an estimator being minimax among all linear affine estimators is also minimax in the set of all estimators. Two illustrative special cases are mentioned, where a generalized least squares estimator and a general ridge or Kuks-Olman estimator turn out to be minimax.     

On estimation of variance components*

In this survey paper the estimation of variance components is given. The least squares approach in variance component estimation is a unifying principle which includes the analysis of variance estimators and the MINQUE. When normality is assumed the maximum likelihood estimators can be used. Many variance component estimators are not permissible because they are not non-negative. The development of non-negative variance component estimators is indicated.     

Small sample properties of the mixed regression estimator

In this paper, the small sample properties of the mixed regression estimator are examined when prior information may be biased and when the ration of the variance of the prior restriction errors to the variance of the sample errors is unknown. The mean square error of the mixed regression estimator is derived, and it is shown that the mixed regression estimator gets dominated by the ordinary least squares estimator in terms of the mean square error as the bias of prior information gets larger.     

Textual Analysis in Real Estate

This paper incorporates text data from MLS listings into a hedonic pricing model. We show that the comments section of the MLS, which is populated by real estate agents who arguably have the most local market knowledge and know what homebuyers value, provides information that improves the performance of both in‐sample and out‐of‐sample pricing estimates. Text is found to decrease pricing error by more than 25%. Information from text is incorporated into a linear model using a tokenization approach. By doing so, the implicit prices for various words and phrases are estimated. The estimation focuses on simultaneous variable selection and estimation for linear models in the presence of a large number of variables using a penalized regression. The LASSO procedure and variants are shown to outperform least‐squares in out‐of‐sample testing. Copyright © 2016 John Wiley & Sons, Ltd.     

Fully modified semiparametric GLS estimation for regressions with nonstationary seasonal regressors

Regression models with seasonally integrated and possibly endogenous regressors and serially correlated regression errors are studied. Spectral decompositions of generalized sums of cross products of regressors and regression errors are used to develop a feasible generalized least squares estimator (FGLSE) which does not require parametric specifications for error processes. Using the FGLSE and following the spirit of “Fully Modified estimation” of Phillips and Hansen (Rev. Econ. Stud. 57 (1990) 99), a fully modified GLSE (FM-GLSE) and inference procedures are constructed. The distribution of the FM-GLSE is shown to be asymptotically a mixed normal distribution which validates standard inference based on the FM-GLSE with normal theory. A Monte-Carlo simulation shows that the FM-GLSE is more efficient than the ordinary least squares estimator (OLSE) in the cases of endogeneity or serial correlation and more efficient than the FM-estimator based on the OLSE in the case of serial correlation.     

The effects of dynamic feedbacks on LS and MM estimator accuracy in panel data models

The finite sample behavior is analyzed of particular least squares (LS) and a range of (generalized) method of moments (MM) estimators in panel data models with individual effects and both a lagged dependent variable regressor and another explanatory variable. The latter may be affected by lagged feedbacks from the dependent variable too. Asymptotic expansions indicate how the order of magnitude of bias of MM estimators tends to increase with the number of moment conditions exploited. They also provide analytic evidence on how the bias of the various estimators depends on the feedbacks and on other model characteristics such as prominence of individual effects and correlation between observed and unobserved heterogeneity. Simulation results corroborate the theoretical findings and reveal that in small samples of models with dynamic feedbacks none of the techniques examined dominates regarding bias and mean squared error over all parametrizations examined.     

Multicollinearity in regression: an efficiency comparison between L<Subscript>p</Subscript>-norm and least squares estimators

Multicollinearity is one of the most important issues in regression analysis, as it produces unstable coefficients’ estimates and makes the standard errors severely inflated. The regression theory is based on specific assumptions concerning the set of error random variables. In particular, when errors are uncorrelated and have a constant variance, the ordinary least squares estimator produces the best estimates among all linear estimators. If, as often happens in reality, these assumptions are not met, other methods might give more efficient estimates and their use is therefore recommendable. In this paper, after reviewing and briefly describing the salient features of the methods, proposed in the literature, to determine and address the multicollinearity problem, we introduce the L_pmin method, based on L_p-norm estimation, an adaptive robust procedure that is used when the residual distribution has deviated from normality. The major advantage of this approach is that it produces more efficient estimates of the model parameters, for different degrees of multicollinearity, than those generated by the ordinary least squares method. A simulation study and a real-data application are also presented, in order to show the better results provided by the L_pmin method in the presence of multicollinearity.     

On the analysis of multivariate growth curves

Growth curve data arise when repeated measurements are observed on a number of individuals with an ordered dimension for occasions. Such data appear frequently in almost all fields in which statistical models are used, for instance in medicine, agriculture and engineering. In medicine, for example, more than one variable is often measured on each occasion. However, analyses are usually based on exploration of repeated measurements of only one variable. The consequence is that the information contained in the between-variables correlation structure will be discarded.  In this study we propose a multivariate model based on the random coefficient regression model for the analysis of growth curve data. Closed-form expressions for the model parameters are derived under the maximum likelihood (ML) and the restricted maximum likelihood (REML) framework. It is shown that in certain situations estimated variances of growth curve parameters are greater for REML. Also a method is proposed for testing general linear hypotheses. One numerical example is provided to illustrate the methods discussed. Received: 22 February 1999