Anomalies in the Foundations of Ridge Regression: Some Clarifications

Several anomalies in the foundations of ridge regression from the perspective of constrained least-square (LS) problems were pointed out in Jensen & Ramirez. Some of these so-called anomalies, attributed to the non-monotonic behaviour of the norm of unconstrained ridge estimators and the consequent lack of sufficiency of Lagrange's principle, are shown to be incorrect. It is noted in this paper that, for a fixed Y , norms of unconstrained ridge estimators corresponding to the given basis are indeed strictly monotone. Furthermore, the conditions for sufficiency of Lagrange's principle are valid for a suitable range of the constraint parameter. The discrepancy arose in the context of one data set due to confusion between estimates of the parameter vector, β , corresponding to different parametrization (choice of bases) and/or constraint norms. In order to avoid such confusion, it is suggested that the parameter β corresponding to each basis be labelled appropriately.     

Ridge regression estimation of the Rotterdam model

The best guesses of unknown coefficients specified in Theil's model of introspection are like predictions and not like de Finetti's prevision and therefore not the values taken by random variables. Constrained least squares procedures can be formulated which are free of these difficulties. The ridge estimator is a simple version of a constrained least squares estimator which can be made operational even when little prior information is available. Our operational ridge estimators are nearly minimax and are not less stable than least squares in the presence of high multicollinearity. Finally, we have presented the ridge estimates for the Rotterdam demand model.     

Estimation and inference under economic restrictions

Estimation of economic relationships often requires imposition of constraints such as positivity or monotonicity on each observation. Methods to impose such constraints, however, vary depending upon the estimation technique employed. We describe a general methodology to impose (observation-specific) constraints for the class of linear regression estimators using a method known as constraint weighted bootstrapping. While this method has received attention in the nonparametric regression literature, we show how it can be applied for both parametric and nonparametric estimators. A benefit of this method is that imposing numerous constraints simultaneously can be performed seamlessly. We apply this method to Norwegian dairy farm data to estimate both unconstrained and constrained parametric and nonparametric models.     

Mean squared error of ridge estimators in logistic regression

It is well known that the maximum likelihood estimator (MLE) is inadmissible when estimating the multidimensional Gaussian location parameter. We show that the verdict is much more subtle for the binary location parameter. We consider this problem in a regression framework by considering a ridge logistic regression (RR) with three alternative ways of shrinking the estimates of the event probabilities. While it is shown that all three variants reduce the mean squared error (MSE) of the MLE, there is at the same time, for every amount of shrinkage, a true value of the location parameter for which we are overshrinking, thus implying the minimaxity of the MLE in this family of estimators. Little shrinkage also always reduces the MSE of individual predictions for all three RR estimators; however, only the naive estimator that shrinks toward 1/2 retains this property for any generalized MSE (GMSE). In contrast, for the two RR estimators that shrink toward the common mean probability, there is always a GMSE for which even a minute amount of shrinkage increases the error. These theoretical results are illustrated on a numerical example. The estimators are also applied to a real data set, and practical implications of our results are discussed.     

Smoothing and Benchmarking for Small Area Estimation

Small area estimation is concerned with methodology for estimating population parameters associated with a geographic area defined by a cross-classification that may also include non-geographic dimensions. In this paper, we develop constrained estimation methods for small area problems: those requiring smoothness with respect to similarity across areas, such as geographic proximity or clustering by covariates, and benchmarking constraints, requiring weighted means of estimates to agree across levels of aggregation. We develop methods for constrained estimation decision theoretically and discuss their geometric interpretation. The constrained estimators are the solutions to tractable optimisation problems and have closed-form solutions. Mean squared errors of the constrained estimators are calculated via bootstrapping. Our approach assumes the Bayes estimator exists and is applicable to any proposed model. In addition, we give special cases of our techniques under certain distributional assumptions. We illustrate the proposed methodology using web-scraped data on Berlin rents aggregated over areas to ensure privacy.     

Estimation of linear models with nequal restrictions

Inequality constrained regression involves the notion of a truncated parameter space, which was studied extensively by Moors (1985). His general results are extended here and applied to linear models. Using the invariance principle, for every observation x a set V_x is defined with the property that estimators taking values in V_x (with positive probability) are inadmissible. One of the main conclusions is that the usual estimators in inequality constrained regression are inadmissible; a method to obtain better estimators is indicated.     

The generalized ridge estimator and improved adjustments for regression parameters

Summary The generalized ridge estimator, which considers generalizations of mean square error, is presented, and a mathematical rule of determining the optimalk-value is discussed. The generalized ridge estimator is examined in comparison with the least squares, the pseudoinverse, theJames-Stein-type shrinkage, and the principal component estimators, especially focusing their attention on improved adjustments for regression coefficients. An alternative estimation approach that better integrates a priori information is noted. Finally, combining the generalized ridge and robust regression methods is suggested.     

Preliminary test ridge regression estimators with student’s <Emphasis Type="Italic">t</Emphasis> errors and conflicting test-statistics

The preliminary test ridge regression estimators (PTRRE) based on the Wald (W), Likelihood Ratio (LR) and Lagrangian Multiplier (LM) tests for estimating the regression parameters has been considered in this paper. Here we consider the multiple regression model with student t error distribution. The bias and the mean square errors (MSE) of the proposed estimators are derived under both null and alternative hypothesis. By studying the MSE criterion, the regions of optimality of the estimators are determined. Under the null hypothesis, the PTRRE based on LM test has the smallest risk followed by the estimators based on LR and W tests. However, the PTRRE based on W test performs the best followed by the LR and LM based estimators when the parameter moves away from the subspace of the restrictions. The conditions of superiority of the proposed estimators for both shrinkage parameter, k and the departure parameter, are provided. Some tables for the maximum and minimum guaranteed efficiency of the proposed estimators have been given, which allows us to determine the optimum level of significance corresponding to the optimum estimator. Finally, we conclude that the estimator based on Wald test dominates the other two estimators in the sense of having highest minimum guaranteed efficiency.     

Preliminary test Liu estimators based on the conflicting W,LR and LM tests in a regression model with multivariate Student-<Emphasis Type="Italic">t</Emphasis> error

In this paper, the problem of estimation of the regression coefficients in a multiple regression model with multivariate Student-t error is considered under the multicollinearity situation when it is suspected that the regression coefficients may be restricted to a linear manifold. The preliminary test Liu estimators (PTLE) based on the Wald, Likelihood ratio (LR) and Lagrangian multiplier (LM) tests are given. The bias and mean square error (MSE) of the proposed estimators are derived and conditions of superiority of these estimators are provided. In particular, we show that in the neighborhood of the null hypothesis, the PTLE based on the LM test has the best performance followed by the estimators based on LR and W tests, while the situation is reversed when the parameter moves away from the manifold of the restriction. Furthermore, the optimum choice of the level of significance is also discussed.     

Shrinkage Estimation Strategies in Generalised Ridge Regression Models: Low/High-Dimension Regime

In this study, we suggest pretest and shrinkage methods based on the generalised ridge regression estimation that is suitable for both multicollinear and high-dimensional problems. We review and develop theoretical results for some of the shrinkage estimators. The relative performance of the shrinkage estimators to some penalty methods is compared and assessed by both simulation and real-data analysis. We show that the suggested methods can be accounted as good competitors to regularisation techniques, by means of a mean squared error of estimation and prediction error. A thorough comparison of pretest and shrinkage estimators based on the maximum likelihood method to the penalty methods. In this paper, we extend the comparison outlined in his work using the least squares method for the generalised ridge regression.     

Ridge regression and its degrees of freedom

For ridge regression the degrees of freedom are commonly calculated by the trace of the matrix that transforms the vector of observations on the dependent variable into the ridge regression estimate of its expected value. For a fixed ridge parameter this is unobjectionable. When the ridge parameter is optimized on the same data, by minimization of the generalized cross validation criterion or Mallows \(\hbox {C}_{L}\) , additional degrees of freedom are used however. We give formulae that take this into account. This allows of a proper assessment of ridge regression in competitions for the best predictor.     

Estimation of spatial autoregressive panel data models with fixed effects

This paper establishes asymptotic properties of quasi-maximum likelihood estimators for SAR panel data models with fixed effects and SAR disturbances. A direct approach is to estimate all the parameters including the fixed effects. Because of the incidental parameter problem, some parameter estimators may be inconsistent or their distributions are not properly centered. We propose an alternative estimation method based on transformation which yields consistent estimators with properly centered distributions. For the model with individual effects only, the direct approach does not yield a consistent estimator of the variance parameter unless T is large, but the estimators for other common parameters are the same as those of the transformation approach. We also consider the estimation of the model with both individual and time effects.     

Asymptotic distributions of some scale estimators in nonlinear models

Often in the robust analysis of regression and time series models there is a need for having a robust scale estimator of a scale parameter of the errors. One often used scale estimator is the median of the absolute residuals s ₁. It is of interest to know its limiting distribution and the consistency rate. Its limiting distribution generally depends on the estimator of the regression and/or autoregressive parameter vector unless the errors are symmetrically distributed around zero. To overcome this difficulty it is then natural to use the median of the absolute differences of pairwise residuals, s ₂, as a scale estimator. This paper derives the asymptotic distributions of these two estimators for a large class of nonlinear regression and autoregressive models when the errors are independent and identically distributed. It is found that the asymptotic distribution of a suitably standardizes s ₂ is free of the initial estimator of the regression/autoregressive parameters. A similar conclusion also holds for s ₁ in linear regression models through the origin and with centered designs, and in linear autoregressive models with zero mean errors.  This paper also investigates the limiting distributions of these estimators in nonlinear regression models with long memory moving average errors. An interesting finding is that if the errors are symmetric around zero, then not only is the limiting distribution of a suitably standardized s ₁ free of the regression estimator, but it is degenerate at zero. On the other hand a similarly standardized s ₂ converges in distribution to a normal distribution, regardless of the errors being symmetric or not. One clear conclusion is that under the symmetry of the long memory moving average errors, the rate of consistency for s ₁ is faster than that of s ₂.     

A generalization of the Binomial distribution based on the dependence ratio

We propose a generalization of the Binomial distribution, called DR‐Binomial, which accommodates dependence among units through a model based on the dependence ratio (Ekholm et al., Biometrika, 82, 1995, 847). Properties of the DR‐Binomial are discussed, and the constraints on its parameter space are studied in detail. Likelihood‐based inference is presented, using both the joint and profile likelihoods; the usefulness of the DR‐Binomial in applications is illustrated on a real dataset displaying negative unit‐dependence, and hence under‐dispersion compared with the Binomial. Although the DR‐Binomial turns out to be a reparameterization of Altham's Additive‐Binomial and Kupper–Haseman's Correlated‐Binomial distribution, we believe its introduction is useful, both in terms of interpretability and mathematical tractability and in terms of generalizability to the Multinomial case.     

Quasi-maximum likelihood estimation for conditional quantiles

In this paper, we construct a new class of estimators for conditional quantiles in possibly misspecified nonlinear models with time series data. Proposed estimators belong to the family of quasi-maximum likelihood estimators (QMLEs) and are based on a new family of densities which we call ‘tick-exponential’. A well-known member of the tick-exponential family is the asymmetric Laplace density, and the corresponding QMLE reduces to the Koenker and Bassett's (Econometrica 46 (1978) 33) nonlinear quantile regression estimator. We derive primitive conditions under which the tick-exponential QMLEs are consistent and asymptotically normally distributed with an asymptotic covariance matrix that accounts for possible conditional quantile model misspecification and which can be consistently estimated by using the tick-exponential scores and Hessian matrix. Despite its non-differentiability, the tick-exponential quasi-likelihood is easy to maximize by using a ‘minimax’ representation not seen in the earlier work on conditional quantile estimation.     

Estimation of location of discontinuity in a density

Here we propose a few estimators of θ, in addition to those studied in Goria (1978), the point of discontinuity of the probability density $$f(x,\theta ) = \frac{1}{{2\Gamma (\alpha )}}e^{ - |x - \theta |} |x - \theta |^{\alpha - 1} ,$$ for $$0< \alpha< 1, - \infty< x< \infty , - \infty< \theta< \infty .$$ We establish the consistency and the optimality of the Bayes and the maximum probability estimators. Despite their nice properties, these estimators are not easy to compute in this case and their effective computation depends on the knowledge of the exponent α. Hence, we propose another class of estimators, dependent upon the spacings of the observations, computable without actual knowledge of the value of α as long as it is known that α < α₀ < 1: we show that these estimators converge at the best possible rate. We further demonstrate, using a modified version of the maximum probability estimator's technique, that the tails of the density do not substantially effect their efficiency. Finally a bivariate family of densities, having a ridge dependent on the parameter θ, is considered and it is shown that this family exhibits features similar to the univariate case, and thus, the necessary modifications of the arguments of the univariate case are utilized for the estimation of θ in this bivariate example.     

Matching and semi-parametric IV estimation, a distance-based measure of migration, and the wages of young men

Our paper estimates the effect of US internal migration on wage growth for young men between their first and second job. Our analysis of migration extends previous research by: (i) exploiting the distance-based measures of migration in the National Longitudinal Surveys of Youth 1979 (NLSY79); (ii) allowing the effect of migration to differ by schooling level and (iii) using propensity score matching to estimate the average treatment effect on the treated (ATET) for movers and (iv) using local average treatment effect (LATE) estimators with covariates to estimate the average treatment effect (ATE) and ATET for compliers.We believe the Conditional Independence Assumption (CIA) is reasonable for our matching estimators since the NLSY79 provides a relatively rich array of variables on which to match. Our matching methods are based on local linear, local cubic, and local linear ridge regressions. Local linear and local ridge regression matching produce relatively similar point estimates and standard errors, while local cubic regression matching badly over-fits the data and provides very noisy estimates.We use the bootstrap to calculate standard errors. Since the validity of the bootstrap has not been investigated for the matching estimators we use, and has been shown to be invalid for nearest neighbor matching estimators, we conduct a Monte Carlo study on the appropriateness of using the bootstrap to calculate standard errors for local linear regression matching. The data generating processes in our Monte Carlo study are relatively rich and calibrated to match our empirical models or to test the sensitivity of our results to the choice of parameter values. The estimated standard errors from the bootstrap are very close to those from the Monte Carlo experiments, which lends support to our using the bootstrap to calculate standard errors in our setting.From the matching estimators we find a significant positive effect of migration on the wage growth of college graduates, and a marginally significant negative effect for high school dropouts. We do not find any significant effects for other educational groups or for the overall sample. Our results are generally robust to changes in the model specification and changes in our distance-based measure of migration. We find that better data matters; if we use a measure of migration based on moving across county lines, we overstate the number of moves, while if we use a measure based on moving across state lines, we understate the number of moves. Further, using either the county or state measures leads to much less precise estimates.We also consider semi-parametric LATE estimators with covariates (Frölich 2007), using two sets of instrumental variables. We precisely estimate the proportion of compliers in our data, but because we have a small number of compliers, we cannot obtain precise LATE estimates.     

An unexpected property of minimax estimation in the relative squared error approach to linear regression analysis

An unexpected property of the relative squared error approach to linear regression analysis is derived: It is shown that an estimator being minimax among all linear affine estimators is also minimax in the set of all estimators. Two illustrative special cases are mentioned, where a generalized least squares estimator and a general ridge or Kuks-Olman estimator turn out to be minimax.     

Spurious Fixed Effects Regression*

This article shows that spurious regression results can occur for a fixed effects model with weak time series variation in the regressor and/or strong time series variation in the regression errors when the first‐differenced and Within‐OLS estimators are used. Asymptotic properties of these estimators and the related t‐tests and model selection criteria are studied by sending the number of cross‐sectional observations to infinity. This article shows that the first‐differenced and Within‐OLS estimators diverge in probability, that the related t‐tests are inconsistent, that R²s converge to zero in probability and that AIC and BIC diverge to ?∞ in probability. The results of the article warn that one should not jump to the use of fixed effects regressions without considering the degree of time series variations in the data.     

Ridge regression revisited

In general ridge (GR) regression p ridge parameters have to be determined, whereas simple ridge regression requires the determination of only one parameter. In a recent textbook on linear regression, Jürgen Gross argues that this constitutes a major complication. However, as we show in this paper, the determination of these p parameters can fairly easily be done. Furthermore, we introduce a generalization of the GR estimator derived by Hemmerle and by Teekens and de Boer. This estimator, which is more conservative, performs better than the Hoerl and Kennard estimator in terms of a weighted quadratic loss criterion.