The Coverage Properties of Confidence Regions After Model Selection

It is very common in applied frequentist ("classical") statistics to carry out a preliminary statistical (i.e. data-based) model selection by, for example, using preliminary hypothesis tests or minimizing AIC. This is usually followed by the inference of interest, using the same data, based on the assumption that the selected model had been given to us  a priori . This assumption is false and it can lead to an inaccurate and misleading inference. We consider the important case that the inference of interest is a confidence region. We review the literature that shows that the resulting confidence regions typically have very poor coverage properties. We also briefly review the closely related literature that describes the coverage properties of prediction intervals after preliminary statistical model selection. A possible motivation for preliminary statistical model selection is a wish to utilize uncertain prior information in the inference of interest. We review the literature in which the aim is to utilize uncertain prior information directly in the construction of confidence regions, without requiring the intermediate step of a preliminary statistical model selection. We also point out this aim as a future direction for research.     

Half‐panel jackknife fixed‐effects estimation of linear panels with weakly exogenous regressors

This paper considers estimation and inference in linear panel regression models with lagged dependent variables and/or other weakly exogenous regressors when N (the cross‐section dimension) is large relative to T (the time series dimension). It allows for fixed and time effects (FE‐TE) and derives a general formula for the bias of the FE‐TE estimator which generalizes the well‐known Nickell bias formula derived for the pure autoregressive dynamic panel data models. It shows that in the presence of weakly exogenous regressors inference based on the FE‐TE estimator will result in size distortions unless N/T is sufficiently small. To deal with the bias and size distortion of the FE‐TE estimator the use of a half‐panel jackknife FE‐TE estimator is considered and its asymptotic distribution is derived. It is shown that the bias of the half‐panel jackknife FE‐TE estimator is of order T^?2, and for valid inference it is only required that N/T³→0, as N,T→∞ jointly. Extension to unbalanced panel data models is also provided. The theoretical results are illustrated with Monte Carlo evidence. It is shown that the FE‐TE estimator can suffer from large size distortions when N>T, with the half‐panel jackknife FE‐TE estimator showing little size distortions. The use of half‐panel jackknife FE‐TE estimator is illustrated with two empirical applications from the literature.     

The emperor's new clothes: a critique of the multivariate t regression model

Zellner (1976) proposed a regression model in which the data vector (or the error vector) is represented as a realization from the multivariate Student t distribution. This model has attracted considerable attention because it seems to broaden the usual Gaussian assumption to allow for heavier-tailed error distributions. A number of results in the literature indicate that the standard inference procedures for the Gaussian model remain appropriate under the broader distributional assumption, leading to claims of robustness of the standard methods. We show that, although mathematically the two models are different, for purposes of statistical inference they are indistinguishable. The empirical implications of the multivariate t model are precisely the same as those of the Gaussian model. Hence the suggestion of a broader distributional representation of the data is spurious, and the claims of robustness are misleading. These conclusions are reached from both frequentist and Bayesian perspectives.     

A note on the uniform distribution on the arcsin points

In his book Pukelsheim [8] pointed out that designs supported at the arcsin points are very efficient for the statistical inference in a polynomial regression model. In this note we determine the canonical moments of a class of distributions which have nearly equal weights at the arcsin points. The class contains theD-optimal arcsin support design and theD ₁-optimal design for a polynomial regression. The results allow explicit representations ofD-, andD ₁-efficiencies of these designs in all polynomial models with a degree less than the number of support points of the design.     

One-stage estimation of the effects of operational conditions and practices on productive performance: asymptotically normal and efficient,root-<Emphasis Type="Italic">n</Emphasis> consistent StoNEZD method

Understanding the effects of operational conditions and practices on productive efficiency can provide valuable economic and managerial insights. The conventional approach is to use a two-stage method where the efficiency estimates are regressed on contextual variables representing the operational conditions. The main problem of the two-stage approach is that it ignores the correlations between inputs and contextual variables. To address this shortcoming, we build on the recently developed regression interpretation of data envelopment analysis (DEA) to develop a new one-stage semi-nonparametric estimator that combines the nonparametric DEA-style frontier with a regression model of the contextual variables. The new method is referred to as stochastic semi-nonparametric envelopment of z variables data (StoNEZD). The StoNEZD estimator for the contextual variables is shown to be statistically consistent under less restrictive assumptions than those required by the two-stage DEA estimator. Further, the StoNEZD estimator is shown to be unbiased, asymptotically efficient, asymptotically normally distributed, and converge at the standard parametric rate of order n ^−1/2. Therefore, the conventional methods of statistical testing and confidence intervals apply for asymptotic inference. Finite sample performance of the proposed estimators is examined through Monte Carlo simulations.     

Confidence Distribution,the Frequentist Distribution Estimator of a Parameter: A Review

In frequentist inference, we commonly use a single point (point estimator) or an interval (confidence interval/“interval estimator”) to estimate a parameter of interest. A very simple question is: Can we also use a distribution function (“distribution estimator”) to estimate a parameter of interest in frequentist inference in the style of a Bayesian posterior? The answer is affirmative, and confidence distribution is a natural choice of such a “distribution estimator”. The concept of a confidence distribution has a long history, and its interpretation has long been fused with fiducial inference. Historically, it has been misconstrued as a fiducial concept, and has not been fully developed in the frequentist framework. In recent years, confidence distribution has attracted a surge of renewed attention, and several developments have highlighted its promising potential as an effective inferential tool. This article reviews recent developments of confidence distributions, along with a modern definition and interpretation of the concept. It includes distributional inference based on confidence distributions and its extensions, optimality issues and their applications. Based on the new developments, the concept of a confidence distribution subsumes and unifies a wide range of examples, from regular parametric (fiducial distribution) examples to bootstrap distributions, significance (p‐value) functions, normalized likelihood functions, and, in some cases, Bayesian priors and posteriors. The discussion is entirely within the school of frequentist inference, with emphasis on applications providing useful statistical inference tools for problems where frequentist methods with good properties were previously unavailable or could not be easily obtained. Although it also draws attention to some of the differences and similarities among frequentist, fiducial and Bayesian approaches, the review is not intended to re‐open the philosophical debate that has lasted more than two hundred years. On the contrary, it is hoped that the article will help bridge the gaps between these different statistical procedures.     

Aspects of statistical analysis in DEA-type frontier models

In Grosskopf (1995) and Banker (1995) different approaches and problems of statistical inference in DEA frontier models are presented. This paper focuses on the basic characteristics of DEA models from a statistical point of view. It arose from comments and discussions on both papers above. The framework of DEA models is deterministic (all the observed points lie on the same side of the frontier), nevertheless a stochastic model can be constructed once a data generating process is defined. So statistical analysis may be performed and sampling properties of DEA estimators can be established. However, practical statistical inference (such as test of hypothesis, confidence intervals) still needs artifacts like the bootstrap to be performed. A consistent bootstrap relies also on a clear definition of the data generating proces and on a consistent estimator of it: The approach of Simar and Wilson (1995) is described. Finally, some trails are proposed for introducing stochastic noise in DEA models, in the spirit of the Kneip-Simar (1995) approach.     

MIS‐SPECIFICATION TESTING IN RETROSPECT

The primary objective of this paper is threefold. First, to undertake a retrospective view of Mis‐Specification (M‐S) testing, going back to the early 20th century, with a view to (i) place it in the broader context of modeling and inference and (ii) bring out some of its special features. Second, to call into question several widely used arguments undermining the importance of M‐S testing in favor of relying on weak probabilistic assumptions in conjunction with generic robustness claims and asymptotic inference. Third, to bring out the crucial role of M‐S testing in securing trustworthy inference results. This is achieved by extending/modifying Fisher's statistical framework with a view to draw a clear line between the modeling and the inference facets of statistical induction. The proposed framework untangles the statistical from the substantive (structural) model and focuses on how to secure the adequacy of the statistical model before probing for substantive adequacy. A case is made for using joint M‐S tests based on custom‐built auxiliary regressions with a view to enhance the effectiveness and reliability of probing for potential statistical misspecifications.     

Bayesian and Frequentist Inference for Ecological Inference: The R×C Case

In this paper we propose Bayesian and frequentist approaches to ecological inference, based on R × C contingency tables, including a covariate. The proposed Bayesian model extends the binomial-beta hierarchical model developed by K ing , R osen and T anner (1999) from the 2×2 case to the R × C case. As in the 2×2 case, the inferential procedure employs Markov chain Monte Carlo (MCMC) methods. As such, the resulting MCMC analysis is rich but computationally intensive. The frequentist approach, based on first moments rather than on the entire likelihood, provides quick inference via nonlinear least-squares, while retaining good frequentist properties. The two approaches are illustrated with simulated data, as well as with real data on voting patterns in Weimar Germany. In the final section of the paper we provide an overview of a range of alternative inferential approaches which trade-off computational intensity for statistical efficiency.     

Sequential control of time series by functionals of kernal-weighted empirical processes under local alternatives

Motivated in part by applications in model selection in statistical genetics and sequential monitoring of financial data, we study an empirical process framework for a class of stopping rules which rely on kernel-weighted averages of past data. We are interested in the asymptotic distribution for time series data and an analysis of the joint influence of the smoothing policy and the alternative defining the deviation from the null model (in-control state). We employ a certain type of local alternative which provides meaningful insights. Our results hold true for short memory processes which satisfy a weak mixing condition. By relying on an empirical process framework we obtain both asymptotic laws for the classical fixed sample design and the sequential monitoring design. As a by-product we establish the asymptotic distribution of the Nadaraya-Watson kernel smoother when the regressors do not get dense as the sample size increases.Acknowledgements The author is grateful to two anonymous referees for their constructive comments, which improved the paper. One referee draws my attention to Lifshits paper. The financial support of the Collaborative Research Centre Reduction of Complexity in Multivariate Data Structures (SFB 475) of the German Research Foundation (DFG) is greatly acknowledged.     

Inferring strategies from observed actions: a nonparametric, binary tree classification approach

This paper introduces a non-parametric binary classification tree approach to inferring unobserved strategies from the observed actions of economic agents. The strategies are in the form of possibly nested if–then statements. We apply our approach to experimental data from the repeated ultimatum game, which was conducted in four different countries by Roth et al. (Am. Econ. Rev. 81 (1991) 1068). We find that strategy inference is consistent with existing inference, provides new explanations for subject behavior, and provides new empirically based hypotheses regarding ultimatum game strategies. We conclude that strategy inference is potentially useful as a complementary method of statistical inference in applied research.     

An alternative quasi likelihood approach,Bayesian analysis and data-based inference for model specification

This paper studies an alternative quasi likelihood approach under possible model misspecification. We derive a filtered likelihood from a given quasi likelihood (QL), called a limited information quasi likelihood (LI-QL), that contains relevant but limited information on the data generation process. Our LI-QL approach, in one hand, extends robustness of the QL approach to inference problems for which the existing approach does not apply. Our study in this paper, on the other hand, builds a bridge between the classical and Bayesian approaches for statistical inference under possible model misspecification. We can establish a large sample correspondence between the classical QL approach and our LI-QL based Bayesian approach. An interesting finding is that the asymptotic distribution of an LI-QL based posterior and that of the corresponding quasi maximum likelihood estimator share the same “sandwich”-type second moment. Based on the LI-QL we can develop inference methods that are useful for practical applications under possible model misspecification. In particular, we can develop the Bayesian counterparts of classical QL methods that carry all the nice features of the latter studied in  White (1982). In addition, we can develop a Bayesian method for analyzing model specification based on an LI-QL.     

Estimation in the generalized Poisson model via robust testing

An estimation method is presented which compromises robust efficiency with computational feasibility in the case of the generalized Poisson model. The formal setup is built on flexible nonparametric extensions of the underlying model. The estimation efficiency is expressed via minimax properties of tests resulting from expansions of estimators. The nonparametric neighborhoods related to the proposed score function are exemplified and a real data case is analysed. The resulting method balances several qualitative features of statistical inference: strong differentiability (asymptotic derivations are more accurate), efficiency and natural model extension (quality of formal basic assumptions).     

Statistical Inference in Nonparametric Frontier Models: The State of the Art

Efficiency scores of firms are measured by their distance to an estimated production frontier. The economic literature proposes several nonparametric frontier estimators based on the idea of enveloping the data (FDH and DEA-type estimators). Many have claimed that FDH and DEA techniques are non-statistical, as opposed to econometric approaches where particular parametric expressions are posited to model the frontier. We can now define a statistical model allowing determination of the statistical properties of the nonparametric estimators in the multi-output and multi-input case. New results provide the asymptotic sampling distribution of the FDH estimator in a multivariate setting and of the DEA estimator in the bivariate case. Sampling distributions may also be approximated by bootstrap distributions in very general situations. Consequently, statistical inference based on DEA/FDH-type estimators is now possible. These techniques allow correction for the bias of the efficiency estimators and estimation of confidence intervals for the efficiency measures. This paper summarizes the results which are now available, and provides a brief guide to the existing literature. Emphasizing the role of hypotheses and inference, we show how the results can be used or adapted for practical purposes.     

Modelling heterogeneity and dynamics in the volatility of individual wages

This paper presents a model for the heterogeneity and dynamics of the conditional mean and conditional variance of individual wages. A bias‐corrected likelihood approach, which reduces the estimation bias to a term of order 1/T², is used for estimation and inference. The small‐sample performance of the proposed estimator is investigated in a Monte Carlo study. The simulation results show that the bias of the maximum likelihood estimator is substantially corrected for designs calibrated to the data used in the empirical analysis, drawn from the PSID. The empirical results show that it is important to account for individual unobserved heterogeneity and dynamics in the variance, and that the latter is driven by job mobility. The model also explains the non‐normality observed in log‐wage data. Copyright © 2010 John Wiley & Sons, Ltd.     

Measurement errors and outliers in seasonal unit root testing

Seasonal and non-seasonal data are frequently observed with noise. For instance, the time series can have irregular abrupt changes and interruptions following as a result of additive or temporary change outliers caused by external circumstances. Equally, the time series can have measurement errors. In this paper we analyse the above types of data irregularities on the behavior of seasonal unit root tests. Outliers and measurement errors can seriously affect seasonal unit root inference and it is shown how the distortion of the tests will depend upon the frequency, magnitude, and persistence of the outliers as well as on the signal to noise ratio associated with measurement errors. Some solutions to the implied inference problems are suggested and shown to work in practice.     

Sampling for Possibilities

This paper views empirical research as a search for illustrations of interesting possibilities which have occurred, and the exploration of the variety of such possibilities in a sample or universe. This leads to a definition of illustrative inference (in contrast to statistical inference), which, we argue, is of considerable importance in many fields of inquiry – ranging from market research and qualitative research in social science, to cosmology. Sometimes, it may be helpful to model illustrative inference quantitatively, so that the size of a sample can be linked to its power (for illustrating possibilities): we outline one model based on probability theory, and another based on a resampling technique.     

Estimation of stochastic frontier models based on multimodel inference

In most empirical studies, once the best model has been selected according to a certain criterion, subsequent analysis is conducted conditionally on the chosen model. In other words, the uncertainty of model selection is ignored once the best model has been chosen. However, the true data-generating process is in general unknown and may not be consistent with the chosen model. In the analysis of productivity and technical efficiencies in the stochastic frontier settings, if the estimated parameters or the predicted efficiencies differ across competing models, then it is risky to base the prediction on the selected model. Buckland et al. (Biometrics 53:603?C618, 1997) have shown that if model selection uncertainty is ignored, the precision of the estimate is likely to be overestimated, the estimated confidence intervals of the parameters are often below the nominal level, and consequently, the prediction may be less accurate than expected. In this paper, we suggest using the model-averaged estimator based on the multimodel inference to estimate stochastic frontier models. The potential advantages of the proposed approach are twofold: incorporating the model selection uncertainty into statistical inference; reducing the model selection bias and variance of the frontier and technical efficiency estimators. The approach is demonstrated empirically via the estimation of an Indian farm data set.     

The association between two random elements: A complete characterization and odds ratio models

For random elements X and Y (e.g. vectors) a complete characterization of their association is given in terms of an odds ratio function. The main result establishes for any odds ratio function and any pre-specified marginals the unique existence of a corresponding joint distribution (the joint density is obtained as a limit of an iterative procedure of marginal fittings). Restricting only the odds ratio function but not the marginals leads to semi-parmetric association models for which statistical inference is available for samples drawn conditionally on either X or Y. Log-bilinear association models for random vectors X and Y are introduced which generalize standard (regression) models by removing restrictions on the marginals. In particular, the logistic regression model is recognized as a log-bilinear association model. And the joint distribution of X and Y is shown to be multivariate normal if and only if both marginals are normal and the association is log-bilinear.Acknowledgements The author thanks both referees for their helpful comments which improved the first draft of the paper.