Concentration behavior of the penalized least squares estimator

Consider the standard nonparametric regression model and take as estimator the penalized least squares function. In this article, we study the trade‐off between closeness to the true function and complexity penalization of the estimator, where complexity is described by a seminorm on a class of functions. First, we present an exponential concentration inequality revealing the concentration behavior of the trade‐off of the penalized least squares estimator around a nonrandom quantity, where such quantity depends on the problem under consideration. Then, under some conditions and for the proper choice of the tuning parameter, we obtain bounds for this nonrandom quantity. We illustrate our results with some examples that include the smoothing splines estimator.     

An exchange algorithm for computing the least quartile difference estimator

We propose an exchange algorithm (EA) for computing the least quartile difference estimate in a multiple linear regression model. Empirical results suggest that the EA is faster and more accurate than the usual p-subset algorithm.     

BALANCED VARIABLE ADDITION IN LINEAR MODELS

This paper studies what happens when we move from a short regression to a long regression in a setting where both regressions are subject to misspecification. In this setup, the least‐squares estimator in the long regression may have larger inconsistency than the least‐squares estimator in the short regression. We provide a simple interpretation for the comparison of the inconsistencies and study under which conditions the additional regressors in the long regression represent a “balanced addition” to the short regression.     

Generalized least squares cross‐validation in kernel density estimation

The kernel density estimation is a popular method in density estimation. The main issue is bandwidth selection, which is a well‐known topic and is still frustrating statisticians. A robust least squares cross‐validation bandwidth is proposed, which significantly improves the classical least squares cross‐validation bandwidth for its variability and undersmoothing, adapts to different kinds of densities, and outperforms the existing bandwidths in statistical literature and software.     

Contamination in linear regression models and its influence on estimators

The consequences of the omission of possibly contaminated observations in a linear regression model for the performance of the ordinary least squares ( LS- ) estimator are discussed. We compare the ordinary L Sestimator with the corresponding 'never pooled' LS -estimator with respect to the matrix-valued mean squared error. Necessary and sufficient conditions are derived for the superiority of an estimator to another one and tests are proposed to check these conditions. Finally the resulting preliminary-test-estimators are investigated.     

The asymptotic behaviour of the estimated generalized least squares method in the linear regression model

Abstract  In the linear regression model the generalized least squares (GLS) method is only applicable if the covariance matrix of the errors is known but for a scalar factor. Otherwise an estimator for this matrix has to be used. Then we speak of the estimated generalized least squares (EGLS) method. In this paper the asymptotic behaviour of both methods is compared. Results are applied to some standard models commonly used in econometrics     

Least squares estimator for regression models with some deterministic time varying parameters

Here we study the least squares estimates in some regression models. We assume that the evolution of the parameter is linearly explosive (i.e. polynomial), or stable (i.e. sinusoidal). We prove the strong consistency, and establish the rate of convergence.     

Consistency and asymptotic normality of least squares estimators in generalized STAR models

Space–time autoregressive (STAR) models, introduced by Cliff and Ord [Spatial autocorrelation (1973) Pioneer, London] are successfully applied in many areas of science, particularly when there is prior information about spatial dependence. These models have significantly fewer parameters than vector autoregressive models, where all information about spatial and time dependence is deduced from the data. A more flexible class of models, generalized STAR models, has been introduced in Borovkova et al. [Proc. 17th Int. Workshop Stat. Model. (2002), Chania, Greece] where the model parameters are allowed to vary per location. This paper establishes strong consistency and asymptotic normality of the least squares estimator in generalized STAR models. These results are obtained under minimal conditions on the sequence of innovations, which are assumed to form a martingale difference array. We investigate the quality of the normal approximation for finite samples by means of a numerical simulation study, and apply a generalized STAR model to a multivariate time series of monthly tea production in west Java, Indonesia.     

Consistency of elementwise-weighted total least squares estimator in a multivariate errors-in-variables model <Emphasis Type="Bold">AX=B</Emphasis>

A multivariate measurement error model AXB is considered. The errors in [A,B] are rowwise independent, but within each row the errors may be correlated. Some of the columns are observed without errors, and in addition the error covariance matrices may differ from row to row. The total covariance structure of the errors is supposed to be known up to a scalar factor. The fully weighted total least squares estimator of X is studied, which in the case of normal errors coincides with the maximum likelihood estimator. We give mild conditions for weak and strong consistency of the estimator, when the number of rows in A increases. The results generalize the conditions of Gallo given for a univariate homoscedastic model (where B is a vector), and extend the conditions of Gleser given for the multivariate homoscedastic model. We derive the objective function for the estimator and propose an iteratively reweighted numerical procedure.Acknowledgements.A. Kukush is supported by a postdoctoral research fellowship of the Belgian office for Scientific, Technical and Cultural Affairs, promoting Scientific and Technical Collaboration with Central and Eastern Europe. S. Van Huffel is a full professor with the Katholieke Universiteit Leuven. This paper presents research results of the Belgian Programme on Interuniversity Poles of Attraction (IUAP Phase V-22), initiated by the Belgian State, Prime Ministers Office-Federal Office for Scientific, Technical and Cultural Affairs, of the Concerted Research Action (GOA) projects of the Flemish Government MEFISTO-666 (Mathematical Engineering for Information and Communication Systems Technology), of the IDO/99/03 project (K.U. Leuven) Predictive computer models for medical classification problems using patient data and expert knowledge, of the FWO projects G.0200.00, G.0078.01 and G.0270.02. The scientific responsibility is assumed by its authors. The authors would like to thank Maria Luisa Rastello and Amedeo Premoli for bringing the EW-TLS problem to their attention. The authors are grateful to two anonymous referees for the valuable comments.     

On the performance of the ordinary least squares method under an error component model

Starting from the one-dimensional results by Wang et al (1994) we consider the performance of the ordinary least squares estimator in comparison to the best linear unbiased estimator under an error component model with random effects in units and time. Upper bounds are derived for the first-order approximation to the difference between both estimators and for the spectral norm of the difference between their dispersion matrices.     

Choosing instrumental variables in conditional moment restriction models

Properties of GMM estimators are sensitive to the choice of instrument. Using many instruments leads to high asymptotic asymptotic efficiency but can cause high bias and/or variance in small samples. In this paper we develop and implement asymptotic mean square error (MSE) based criteria for instrument selection in estimation of conditional moment restriction models. The models we consider include various nonlinear simultaneous equations models with unknown heteroskedasticity. We develop moment selection criteria for the familiar two-step optimal GMM estimator (GMM), a bias corrected version, and generalized empirical likelihood estimators (GEL), that include the continuous updating estimator (CUE) as a special case. We also find that the CUE has lower higher-order variance than the bias-corrected GMM estimator, and that the higher-order efficiency of other GEL estimators depends on conditional kurtosis of the moments.     

Nonparametric regression function estimation using interaction least squares splines and comlexity regularization

Let (X, Y) be a pair of random variables withsupp(X)⊆[0,1] ^l andEY ²<∞. Letm ^* be the best approximation of the regression function of (X, Y) by sums of functions of at mostd variables (1≤d≤l). Estimation ofm ^* from i.i.d. data is considered. For the estimation interaction least squares splines, which are defined as sums of polynomial tensor product splines of at mostd variables, are used. The knot sequences of the tensor product splines are chosen equidistant. Complexity regularization is used to choose the number of the knots and the degree of the splines automatically using only the given data. Without any additional condition on the distribution of (X, Y) the weak and strongL ₂-consistency of the estimate is shown. Furthermore, for everyp≥1 and every distribution of (X, Y) withsupp(X)⊆[0,1] ^l ,y bounded andm ^* p-smooth, the integrated squared error of the estimate achieves up to a logarithmic factor the (optimal) rate      

The exact moments of the least squares estimator for the autoregressive model

Exact mean and variance of the least squares estimate of the stationary first-order autoregressive coefficient, i.e., β in y_t=α+βx_t+u_t are evaluated algebraically as well as numerically. It turns out that the least squares estimate is seriously biased for the sample of two-digits sizes typically dealt with in econometrics if the mean of the process is unknown, i.e., if the equation has a non-zero intercept (α≠0). Kendall's approximation to the mean and Barlett's approximation to the variance are shown to be fairly good. Also, our numerical results confirm Orcutt and Winokur's (Econometrica, Vol. 37) based on Monte Carlo experiments.     

Robust normal reference bandwidth for kernel density estimation

Bandwidth selection is the main problem of kernel density estimation, the most popular method of density estimation. The classical normal reference bandwidth usually oversmoothes the density estimate. The existing hi-tech bandwidths have computational problems (even may not exist) and are not robust against outliers in the sample. A highly robust normal reference bandwidth is proposed, which adapts to different types of densities.     

Another look at the naive estimator in a regression model

We consider the linear regression model where only a particular linear function of the dependent variables is observed, Stahlecker and Schmidt (1987) proposed a naive least squares (LS) estimator for regression coefficients in such a case. In this note we represent their estimator as a general ridge estimator. This observation leads to a view different from the previous work and provides an easy way of obtaining many important properties of the naive LS estimator. Our approach also gives some insight into the relationship between the naive LS estimator and the generalized least squares estimator.     

Out-of-sample forecasts of China's economic growth and inflation using rolling weighted least squares

Macroeconomic forecasting in China is essential for the government to take proper policy decisions on government expenditure and money supply, among other matters. The existing literature on forecasting Chinas macroeconomic variables is unclear on the crucial issue of how to choose an optimal window to estimate parameters with rolling out-of-sample forecasts. This study fills this gap in forecasting economic growth and inflation in China, by using the rolling weighted least squares (WLS) with the practically feasible cross-validation (CV) procedure of Hong et al. (2018) to choose an optimal estimation window. We undertake an empirical analysis of monthly data on up to 30 candidate indicators (mainly asset prices) for a span of 17 years (2000–2017). It is documented that the forecasting performance of rolling estimation is sensitive to the selection of rolling windows. The empirical analysis shows that the rolling WLS with the CV-based rolling window outperforms other rolling methods on univariate regressions in most cases. One possible explanation for this is that these macroeconomic variables often suffer from structural changes due to changes in institutional reforms, policies, crises, and other factors. Furthermore, we find that, in most cases, asset prices are key variables for forecasting macroeconomic variables, especially output growth rate.     

Improved forecasting of autoregressive series by weighted least squares approximate REML estimation

Restricted maximum likelihood (REML) estimation has recently been shown to provide less biased estimates in autoregressive series. A simple weighted least squares approximate REML procedure has been developed that is particularly useful for vector autoregressive processes. Here, we compare the forecasts of such processes using both the standard ordinary least squares (OLS) estimates and the new approximate REML estimates. Forecasts based on the approximate REML estimates are found to provide a significant improvement over those obtained using the standard OLS estimates.     

Using partial least squares in operations management research: A practical guideline and summary of past research

The partial least squares (PLS) approach to structural equation modeling (SEM) has been widely adopted in business research fields such as information systems, consumer behavior, and marketing. The use of PLS in the field of operations management is also growing. However, questions still exist among some operations management researchers regarding whether and how PLS should be used. To address these questions, our study provides a practical guideline for using PLS and uses examples from the operations management literature to demonstrate how the specific points in this guideline can be applied. In addition, our study reviews and summarizes the use of PLS in the recent operations management literature according to our guideline. The main contribution of this study is to present a practical guideline for evaluating and using PLS that is tailored to the operations management field.     

Performance of empirical Bayes estimators of random coefficients in multilevel analysis: Some results for the random intercept-only model

For a multilevel model with two levels and only a random intercept, the quality of different estimators of the random intercept is examined. Analytical results are given for the marginal model interpretation where negative estimates of the variance components are allowed for. Except for four or five level-2 units, the Empirical Bayes Estimator (EBE) has a lower average Bayes risk than the Ordinary Least Squares Estimator (OLSE). The EBEs based on restricted maximum likelihood (REML) estimators of the variance components have a lower Bayes risk than the EBEs based on maximum likelihood (ML) estimators. For the hierarchical model interpretation, where estimates of the variance components are restricted being positive, Monte Carlo simulations were done. In this case the EBE has a lower average Bayes risk than the OLSE, also for four or five level-2 units. For large numbers of level-1 (30) or level-2 units (100), the performances of REML-based and ML-based EBEs are comparable. For small numbers of level-1 (10) and level-2 units (25), the REML-based EBEs have a lower Bayes risk than ML-based EBEs only for high intraclass correlations (0.5).     

Analysis of variance on indirectly measured variables

Abstract  When observations from a normal distribution can only be obtained indirectly by counting the number of subjects responding to a previously chosen dose, parameter estimates can be obtained by using probit analysis. Well-known is the maximum likelihood technique of parameter estimation, less known is the approach by weighted least squares. The latter approach is followed to compare the parameters of several normal distributions by testing their equality, in analogy with the analysis of variance. A practical situation gave rise to this study and it is worked out at the end of the paper.