Proportional Odds Models with High‐Dimensional Data Structure

The proportional odds model is the most widely used model when the response has ordered categories. In the case of high‐dimensional predictor structure, the common maximum likelihood approach typically fails when all predictors are included. A boosting technique pomBoost is proposed to fit the model by implicitly selecting the influential predictors. The approach distinguishes between metric and categorical predictors. In the case of categorical predictors, where each predictor relates to a set of parameters, the objective is to select simultaneously all the associated parameters. In addition, the approach distinguishes between nominal and ordinal predictors. In the case of ordinal predictors, the proposed technique uses the ordering of the ordinal predictors by penalizing the difference between the parameters of adjacent categories. The technique has also a provision to consider some mandatory predictors (if any) that must be part of the final sparse model. The performance of the proposed boosting algorithm is evaluated in a simulation study and applications with respect to mean squared error and prediction error. Hit rates and false alarm rates are used to judge the performance of pomBoost for selection of the relevant predictors.     

Semiparametric quantile averaging in the presence of high-dimensional predictors

The paper proposes a method for forecasting conditional quantiles. In practice, one often does not know the “true” structure of the underlying conditional quantile function, and in addition, we may have a large number of predictors. Focusing on such cases, we introduce a flexible and practical framework based on penalized high-dimensional quantile averaging. In addition to prediction, we show that the proposed method can also serve as a predictor selector. We conduct extensive simulation experiments to asses its prediction and variable selection performances for nonlinear and linear time series model designs. In terms of predictor selection, the approach tends to select the true set of predictors with minimal false positives. With respect to prediction accuracy, the method competes well even with the benchmark/oracle methods that know one or more aspects of the underlying quantile regression model. We further illustrate the merit of the proposed method by providing an application to the out-of-sample forecasting of U.S. core inflation using a large set of monthly macroeconomic variables based on FRED-MD database. The application offers several empirical findings.     

Penalized Regression with Ordinal Predictors

Ordered categorial predictors are a common case in regression modelling. In contrast to the case of ordinal response variables, ordinal predictors have been largely neglected in the literature. In this paper, existing methods are reviewed and the use of penalized regression techniques is proposed. Based on dummy coding two types of penalization are explicitly developed; the first imposes a difference penalty, the second is a ridge type refitting procedure. Also a Bayesian motivation is provided. The concept is generalized to the case of non-normal outcomes within the framework of generalized linear models by applying penalized likelihood estimation. Simulation studies and real world data serve for illustration and to compare the approaches to methods often seen in practice, namely simple linear regression on the group labels and pure dummy coding. Especially the proposed difference penalty turns out to be highly competitive.     

Modeling Clustered Ordered Categorical Data: A Survey

This article surveys various strategies for modeling ordered categorical (ordinal) response variables when the data have some type of clustering, extending a similar survey for binary data by Pendergast, Gange, Newton, Lindstrom, Palta & Fisher (1996). An important special case is when repeated measurement occurs at various occasions for each subject, such as in longitudinal studies. A much greater variety of models and fitting methods are available than when a similar survey for repeated ordinal response data was prepared a decade ago (Agresti, 1989). The primary emphasis of the review is on two classes of models, marginal models for which effects are averaged over all clusters at particular levels of predictors, and cluster-specific models for which effects apply at the cluster level. We present the two types of models in the ordinal context, review the literature for each, and discuss connections between them. Then, we summarize some alternative modeling approaches and ways of estimating parameters, including a Bayesian approach. We also discuss applications and areas likely to be popular for future research, such as ways of handling missing data and ways of modeling agreement and evaluating the accuracy of diagnostic tests. Finally, we review the current availability of software for using the methods discussed in this article.     

Consistent order selection with strongly dependent data and its application to efficient estimation

Order selection based on criteria by Akaike (IEEE Trans. Automat. Control AC-19 (1974) 716), AIC, Schwarz (Ann. Stat. (1978) 461), BIC or Hannan and Quinn's (J. R. Stat. Soc. Ser. B (1979) 190) HIC is often applied in empirical examples. They have been used in the context of order selection of weakly dependent ARMA models, AR models with unit or explosive roots and in the context of regression or distributed lag regression models for weakly dependent data. On the other hand, it has been observed that data exhibits the so-called strong dependence in many areas. Because the interest to this type of data, our main objective in this paper is to examine order selection for a distributed lag regression model that covers in a unified form weak and strong dependence. To that end, and because the possible adverse properties of the aforementioned criteria, we propose a criterion function based on the decomposition of the variance of the innovations of the model in terms of their frequency components. Assuming that the order of the model is finite, say p₀, we show that the proposed criterion consistently estimates p₀. In addition, we show that adaptive estimation for the parameters of the model is possible without knowledge of p₀. Finally, a small Monte-Carlo experiment is included to illustrate the finite sample performance of the proposed criterion.     

Combining information for prediction in linear regression

This article deals with the prediction problem in linear regression where the measurements are obtained using k different devices or collected from k different independent sources. For the case of k=2, a Graybill-Deal type combined estimtor for the regression parameters is shown to dominate the individual least squares estimators under the covariance criterion. Two predictors ŷ _c and ŷ _p are proposed. ŷ _c is based on a combined estimator of the regression coefficient vector, and ŷ _p is obtained by combining the individual predictors from different models. Prediction mean square errors of both predictors are derived. It is shown that the predictor ŷ _p is better than the individual predictors for k≥2 and the predictor ŷ _c is better than the individual predictors for k=2. Numerical comparison between ŷ _c and ŷ _p shows that the former is superior to the latter for the case k=2.     

The nonrandom selection of don't knows in binary and ordinal responses: Corrections with the bivariate probit model with sample selection

On social surveysdon't knows are a common answer to attitudinal questions, which often have binary or ordinal response categories.Don't knows can be nonrandomly selected according to certain demographic or socioeconomic characteristics of the respondent. To model the sample selection and correct for its bias, this paper discusses two types of bivariate models —binary-probit and the ordinal probit model with sample selection. The difference between parameter estimates and predicted probabilities from the analysis modelling the sample selection bias ofdon't knows and those from the analysis not modellingdon't knows is emphasized. Two empirical examples using the 1989 General Social Survey data demonstrate the necessity to correct for the bias in the nonrandom selection ofdon't knows for binary and ordinal attitudinal response variables. A replication of the analyses using the 1990 and 1991 General Social Survey data helps demonstrate the reliability of the sample selection bias ofdon't knows.     

Forecasting crude oil market volatility using variable selection and common factor

This paper aims to improve the predictability of aggregate oil market volatility with a substantially large macroeconomic database, including 127 macro variables. To this end, we use machine learning from both the variable selection (VS) and common factor (i.e., dimension reduction) perspectives. We first use the lasso, elastic net (ENet), and two conventional supervised learning approaches based on the significance level of predictors’ regression coefficients and the incremental R-square to select useful predictors relevant to forecasting oil market volatility. We then rely on the principal component analysis (PCA) to extract a common factor from the selected predictors. Finally, we augment the autoregression (AR) benchmark model by including the supervised PCA common index. Our empirical results show that the supervised PCA regression model can successfully predict oil market volatility both in-sample and out-of-sample. Also, the recommended models can yield forecasting gains in both statistical and economic perspectives. We further shed light on the nature of VS over time. In particular, option-implied volatility is always the most powerful predictor.     

Maximum likelihood estimation based on the Laplace approximation for p2 network regression models

The class of p₂ models is suitable for modeling binary relation data in social network analysis. A p₂ model is essentially a regression model for bivariate binary responses, featuring within‐dyad dependence and correlated crossed random effects to represent heterogeneity of actors. Despite some desirable properties, these models are used less frequently in empirical applications than other models for network data. A possible reason for this is due to the limited possibilities for this model for accounting for (and explicitly modeling) structural dependence beyond the dyad as can be done in exponential random graph models. Another motive, however, may lie in the computational difficulties existing to estimate such models by means of the methods proposed in the literature, such as joint maximization methods and Bayesian methods. The aim of this article is to investigate maximum likelihood estimation based on the Laplace approximation approach, that can be refined by importance sampling. Practical implementation of such methods can be performed in an efficient manner, and the article provides details on a software implementation using R . Numerical examples and simulation studies illustrate the methodology.     

Copula Gaussian graphical models with penalized ascent Monte Carlo EM algorithm

Typical data that arise from surveys, experiments, and observational studies include continuous and discrete variables. In this article, we study the interdependence among a mixed (continuous, count, ordered categorical, and binary) set of variables via graphical models. We propose an ?₁‐penalized extended rank likelihood with an ascent Monte Carlo expectation maximization approach for the copula Gaussian graphical models and establish near conditional independence relations and zero elements of a precision matrix. In particular, we focus on high‐dimensional inference where the number of observations are in the same order or less than the number of variables under consideration. To illustrate how to infer networks for mixed variables through conditional independence, we consider two datasets: one in the area of sports and the other concerning breast cancer.     

Some characterizations of distributions by regression models for ordinal response data

Summary Let a random variableX be classified intok classes. By doing so, a new random variable is obtained, measured on ordinal scale. If this variable is a response variable in certain regression models for ordinal response data, the distribution ofX is characterized by the models. In this paper, characterizations of the distribution ofX by the proportional odds model and the proportional hazards model are given.     

The Ordinal Controversy Revisited

We present a discussion of the different dimensions of the ongoing controversy about the analysis of ordinal variables. The source of this controversy is traced to the earliest possible stage, measurement theory. Three major approaches in analyzing ordinal variables, called the non-parametric, the parametric, and the underlying variable approach, are identified and the merits and drawbacks of each of these approaches are pointed out. We show that the controversy on the exact definition of an ordinal variable causes problems with regard to defining ordinal association, and therefore to the interpretation of many recently designed models for ordinal variables, e.g., structure equation models using polychoric correlations, latent class models and ordinal response models. We conclude that the discussion with regard to ordinal variable modeling can only be fruitful if one makes a distinction between different types of ordinal variables. Five types of ordinal variables were identified. The problems concerning the analysis of these five types of ordinal variables are solved in some cases and remain a problem for others.     

Hidden group patterns in democracy developments: Bayesian inference for grouped heterogeneity

We propose a nonparametric Bayesian approach to estimate time‐varying grouped patterns of heterogeneity in linear panel data models. Unlike the classical approach in Bonhomme and Manresa (Econometrica, 2015, 83, 1147–1184), our approach can accommodate selection of the optimal number of groups and model estimation jointly, and also be readily extended to quantify uncertainties in the estimated group structure. Our proposed approach performs well in Monte Carlo simulations. Using our approach, we successfully replicate the estimated relationship between income and democracy in Bonhomme and Manresa and the group characteristics when we use the same number of groups. Furthermore, we find that the optimal number of groups could depend on model specifications on heteroskedasticity and discuss ways to choose models in practice.     

Efficient big data model selection with applications to fraud detection

As the volume and complexity of data continues to grow, more attention is being focused on solving so-called big data problems. One field where this focus is pertinent is credit card fraud detection. Model selection approaches can identify key predictors for preventing fraud. Stagewise Selection is a classic model selection technique that has experienced a revitalized interest due to its computational simplicity and flexibility. Over a sequence of simple learning steps, stagewise techniques build a sequence of candidate models that is less greedy than the stepwise approach.This paper introduces a new stochastic stagewise technique that integrates a sub-sampling approach into the stagewise framework, yielding a simple tool for model selection when working with big data. Simulation studies demonstrate the proposed technique offers a reasonable trade off between computational cost and predictive performance. We apply the proposed approach to synthetic credit card fraud data to demonstrate the technique’s application.     

Tuning parameter selection in penalized generalized linear models for discrete data

In recent years, we have seen an increased interest in the penalized likelihood methodology, which can be efficiently used for shrinkage and selection purposes. This strategy can also result in unbiased, sparse, and continuous estimators. However, the performance of the penalized likelihood approach depends on the proper choice of the regularization parameter. Therefore, it is important to select it appropriately. To this end, the generalized cross‐validation method is commonly used. In this article, we firstly propose new estimates of the norm of the error in the generalized linear models framework, through the use of Kantorovich inequalities. Then these estimates are used in order to derive a tuning parameter selector in penalized generalized linear models. The proposed method does not depend on resampling as the standard methods and therefore results in a considerable gain in computational time while producing improved results. A thorough simulation study is conducted to support theoretical findings; and a comparison of the penalized methods with the L₁, the hard thresholding, and the smoothly clipped absolute deviation penalty functions is performed, for the cases of penalized Logistic regression and penalized Poisson regression. A real data example is being analyzed, and a discussion follows. © 2014 The Authors. Statistica Neerlandica © 2014 VVS.     

Computing marginal likelihoods from a single MCMC output

In this article, we propose new Monte Carlo methods for computing a single marginal likelihood or several marginal likelihoods for the purpose of Bayesian model comparisons. The methods are motivated by Bayesian variable selection, in which the marginal likelihoods for all subset variable models are required to compute. The proposed estimates use only a single Markov chain Monte Carlo (MCMC) output from the joint posterior distribution and it does not require the specific structure or the form of the MCMC sampling algorithm that is used to generate the MCMC sample to be known. The theoretical properties of the proposed method are examined in detail. The applicability and usefulness of the proposed method are demonstrated via ordinal data probit regression models. A real dataset involving ordinal outcomes is used to further illustrate the proposed methodology.     

SCAD‐penalized quantile regression for high‐dimensional data analysis and variable selection

The present penalized quantile variable selection methods are only applicable to finite number of predictors or do not have oracle property associated with estimator. This technique is considered as an alternative to ordinary least squares regression in case of the outliers and the heavy‐tailed errors existing in linear models. The variable selection through quantile regression with diverging number of parameters is investigated in this paper. The convergence rate of estimator with smoothly clipped absolute deviation penalty function is also studied. Moreover, the oracle property with proper selection of tuning parameter for quantile regression under certain regularity conditions is also established. In addition, the rank correlation screening method is used to accommodate ultra‐high dimensional data settings. Monte Carlo simulations demonstrate finite performance of the proposed estimator. The results of real data reveal that this approach provides substantially more information as compared with ordinary least squares, conventional quantile regression, and quantile lasso.     

Selection correction and sensitivity analysis for ordered treatment effect on count response

In estimating the effect of an ordered treatment τ on a count response y with an observational data where τ is self‐selected (not randomized), observed variables x and unobserved variables ε can be unbalanced across the control group (τ = 0) and the treatment groups (τ = 1, …, J). While the imbalance in x causes ‘overt bias’ which can be removed by controlling for x, the imbalance in ε causes ‘covert (hidden or selection) bias’ which cannot be easily removed. This paper makes three contributions. First, a proper counter‐factual causal framework for ordered treatment effect on count response is set up. Second, with no plausible instrument available for τ, a selection correction approach is proposed for the hidden bias. Third, a nonparametric sensitivity analysis is proposed where the treatment effect is nonparametrically estimated under no hidden bias first, and then a sensitivity analysis is conducted to see how sensitive the nonparametric estimate is to the assumption of no hidden bias. The analytic framework is applied to data from the Health and Retirement Study: the treatment is ordered exercise levels in five categories and the response is doctor office visits per year. The selection correction approach yields very large effects, which are however ruled out by the nonparametric sensitivity analysis. This finding suggests a good deal of caution in using selection correction approaches. Copyright © 2004 John Wiley & Sons, Ltd.     

A Survey of L1 Regression

L₁ regularization, or regularization with an L₁ penalty, is a popular idea in statistics and machine learning. This paper reviews the concept and application of L₁ regularization for regression. It is not our aim to present a comprehensive list of the utilities of the L₁ penalty in the regression setting. Rather, we focus on what we believe is the set of most representative uses of this regularization technique, which we describe in some detail. Thus, we deal with a number of L₁‐regularized methods for linear regression, generalized linear models, and time series analysis. Although this review targets practice rather than theory, we do give some theoretical details about L₁‐penalized linear regression, usually referred to as the least absolute shrinkage and selection operator (lasso).     

Penalized Quantile Regression with Semiparametric Correlated Effects: An Application with Heterogeneous Preferences

This paper proposes new ?₁‐penalized quantile regression estimators for panel data, which explicitly allows for individual heterogeneity associated with covariates. Existing fixed‐effects estimators can potentially suffer from three limitations which are overcome by the proposed approach: (i) incidental parameters bias in nonlinear models with large N and small T ; (ii) lack of efficiency; and (iii) inability to estimate the effects of time‐invariant regressors. We conduct Monte Carlo simulations to assess the small‐sample performance of the new estimators and provide comparisons of new and existing penalized estimators in terms of quadratic loss. We apply the technique to an empirical example of the estimation of consumer preferences for nutrients from a demand model using a large transaction‐level dataset of household food purchases. We show that preferences for nutrients vary across the conditional distribution of expenditure and across genders, and emphasize the importance of fully capturing consumer heterogeneity in demand modeling. Copyright © 2016 John Wiley & Sons, Ltd.