Selecting Adaptive Survey Design Strata with Partial R‐indicators

Recent survey literature shows an increasing interest in survey designs that adapt data collection to characteristics of the survey target population. Given a specified quality objective function, the designs attempt to find an optimal balance between quality and costs. Finding the optimal balance may not be straightforward as corresponding optimisation problems are often highly non‐linear and non‐convex. In this paper, we discuss how to choose strata in such designs and how to allocate these strata in a sequential design with two phases. We use partial R‐indicators to build profiles of the data units where more or less attention is required in the data collection. In allocating cases, we look at two extremes: surveys that are run only once, or infrequent, and surveys that are run continuously. We demonstrate the impact of the sample size in a simulation study and provide an application to a real survey, the Dutch Crime Victimisation Survey.     

Measuring the Measurable: Why can’t we Agree on the Number of Telecommuters in the U.S.?

Using telecommuting as a case study, we demonstrate that definitions, measurement instruments, sampling and sometimes vested interests affect the quality and utility even of seemingly objective and “measurable” data. Little consensus exists with respect to the definition of telecommuting, or to possible distinctions from related terms such as teleworking. Such a consensus is unlikely, since the “best” definition of telecommuting depends on one’s point of reference and purpose. However, differing definitions confound efforts to measure the amount of telecommuting and how it is changing over time. This paper evaluates estimates of the amounts of telecommuting occurring in the U.S. obtained from several different sources: the U.S. Census, the American Housing Survey, several Work at Home supplements to the Current Population Survey, a series of market research surveys, and the trade association-sponsored Telework America surveys. Many of the issues raised here are transferable to other contexts, and indirectly serve as suggestions for improving data collection in the future.     

Evaluating,Comparing, Monitoring,and Improving Representativeness of Survey Response Through R‐Indicators and Partial R‐Indicators

Non‐response is a common source of error in many surveys. Because surveys often are costly instruments, quality‐cost trade‐offs play a continuing role in the design and analysis of surveys. The advances of telephone, computers, and Internet all had and still have considerable impact on the design of surveys. Recently, a strong focus on methods for survey data collection monitoring and tailoring has emerged as a new paradigm to efficiently reduce non‐response error. Paradata and adaptive survey designs are key words in these new developments. Prerequisites to evaluating, comparing, monitoring, and improving quality of survey response are a conceptual framework for representative survey response, indicators to measure deviations thereof, and indicators to identify subpopulations that need increased effort. In this paper, we present an overview of representativeness indicators or R‐indicators that are fit for these purposes. We give several examples and provide guidelines for their use in practice.     

Multiple Visits and Data Quality in Household Surveys

In order to increase data quality some household surveys visit the respondent households several times to estimate one measure of consumption. For example, in Ghanaian Living Standards Measurement surveys, households are visited up to 10 times over a period of 1 month. I find strong evidence for conditioning effects as a result of this approach: In the Ghanaian data the estimated level of consumption is a function of the number of prior visits, with consumption being highest in the earlier survey visits. Telescoping (perceiving events as being more recent than they are) or seasonality (first‐of‐the‐month effects) cannot explain the observed pattern. To study whether earlier or later survey visits are of higher quality, I employ a strategy based on Benford's law. Results suggest that the consumption data from earlier survey visits are of higher quality than data from later visits. The findings have implications for the value of additional visits in household surveys, and also shed light on possible measurement problems in high‐frequency panels. They add to a recent literature on measurement errors in consumption surveys (Beegle et al., 2012 , Gibson et al., 2015 ), and complement findings by Zwane et al. ( 2011 ) regarding the effect of surveys on subsequent behaviour.     

Different approaches to measure ego-centered social support networks: a meta-analysis

Survey indicators of social networks usually measure a certain function of social networks, for example exchange of social support. Social support is a multidimensional construct. The most comprehensive definition distinguishes among sources of social support (social support networks), supportive acts and appraisal of given support. Generally, two main hypotheses can be given with regard to the role social support plays in quality of life of individuals: that social support is beneficial as such (main effects), or that social support is beneficial at occasions of stressful events (buffer effect). In this paper we are dealing with survey measurement of ego-centered social support networks. Three methods to social network measurement are compared: the name generator method, the role generator method and the event-related approach. In a meta-analysis of several studies done on convenient quota samples the effects of method, type of calculation, response format and limitation of support providers on network composition indicators are studied.     

Imputation: Methods, Simulation Experiments and Practical Examples

When conducting surveys, two kinds of nonresponse may cause incomplete data files: unit nonresponse (complete nonresponse) and item nonresponse (partial nonresponse). The selectivity of the unit nonresponse is often corrected for. Various imputation techniques can be used for the missing values because of item nonresponse. Several of these imputation techniques are discussed in this report. One is the hot deck imputation. This paper describes two simulation experiments of the hot deck method. In the first study, data are randomly generated, and various percentages of missing values are then non-randomly'added'to the data. The hot deck method is used to reconstruct the data in this Monte Carlo experiment. The performance of the method is evaluated for the means, standard deviations, and correlation coefficients and compared with the available case method. In the second study, the quality of an imputation method is studied by running a simulation experiment. A selection of the data of the Dutch Housing Demand Survey is perturbed by leaving out specific values on a variable. Again hot deck imputations are used to reconstruct the data. The imputations are then compared with the true values. In both experiments the conclusion is that the hot deck method generally performs better than the available case method. This paper also deals with the questions which variables should be imputed and what the duration of the imputation process is. Finally the theory is illustrated by the imputation approaches of the Dutch Housing Demand Survey, the European Community Household Panel Survey (ECHP) and the new Dutch Structure of Earnings Survey (SES). These examples illustrate the levels of missing data that can be experienced in such surveys and the practical problems associated with choosing an appropriate imputation strategy for key items from each survey.     

Estimation Methods and Related Systems at Statistics Canada

This paper provides an overview of research in estimation techniques, their application, and the development of generalized estimation systems at Statistics Canada. In Canada, the demand for more detailed and better quality cross-sectional data related to various sodo-economic issues has increased significantly in recent years. Also, there has been increasing interest in longitudinal data to better understand and interpret the relationships among variables, necessitating the implementation of a number of large scale panel surveys by Statistics Canada. The paper briefly discusses estimation for longitudinal data and a weighting approach developed for cross-sectional data Prom these surveys. For cross-sectional household and business surveys, as well as the census of population, appropriate dibration estimators developed for each situation are briefly discussed. In addition, regression composite estimation, a method developed to improve the quality of cross-sectional estimates from rotating panel surveys such as the Canadian Labour Force Survey, is presented. With regard to more detailed cross-sectional estimates at sub-provincial levels, different approaches to small area estimation developed for various programs are also presented. We SUmmarize the various modules developed lor the GeneraIiized Ektimation System. important new developments within the system include two-phase estimation as well as the estimation of variance for a number of imputation procedures. We briefly review the status of current estimation research on selected topics as well as the direction of future research.     

Interpreting the results of studies using latent variable models to assess data quality: an empirical example using confirmatory factor analysis

Latent variable models (LVMs) offer one route to examine the quality of data collected in surveys. The possibility exists that individuals equivalent in their true level of a construct or variable being measured are unlikely to have equivalent observed responses as a function of an extraneous variable, e.g., group membership. This potential is labeled here as differential item functioning (DIF). Survey methods generally considers measurement bias to be estimators that do no not accurately reflect true values. DIF may be thought of as differential measurement bias, i.e., measurement bias conditional on group membership. As a function of group membership, the degree, amount, or type of measurement bias changes. DIF has the potential to negatively affect the quality of data. LVMs, e.g., confirmatory factor analysis (CFA), offer one tool to assess DIF. However, few published examples exist in the survey research field and training in the interpretation of these models is lacking. The purpose of the current paper is to describe CFA sufficiently for interpretive purposes and demonstrate an empirical application of CFA to assess survey data quality to provide further interpretive guidance. References are provided for analysts wishing to conduct analyses of this type.     

Towards a flexible online mixed method design with a feedback loop

The notion of mixed methods design relates to the research studies that combine qualitative and quantitative approaches. However, most of these studies are tailored to specific research problem in a particular study and are typically limited to a fixed sequence of qualitative and quantitative approaches (e.g. qualitative interviews followed by a survey or vice-versa). This limitation historically arises from time, cost and logistic restrictions. As an alternative, we develop an general extension of fixed mixed method design by introducing a flexible feedback-loop so that several phases can be combined in a flexible order. In practice, such designs are now increasingly feasible within an information-communication technology environment, where online respondents are readily available for immediate participation. An online experiment combining interactive series of web surveys and in-depth web interviews was performed to compare this approach with standard two-phase designs involving mixed methods. The costs, timing, quality of finiding and experiences of researchers were systematically evaluated. In summary, the proposed approach proved to be very beneficial.     

Testing for measurement equivalence of human values across online and paper-and-pencil surveys

The following study investigates the measurement equivalence of an online and paper-and-pencil (PAP) survey of human values. For this purpose, a total of 250 respondents completed the 21-item version of the Portrait Value Questionnaire either online (n = 125) or by PAP (n = 125). This questionnaire was developed by Schwartz (Advances in experimental social psychology, Academic Press, New York, 1992) and has been included in the European Social Survey since 2002 to test his theory of basic human values. Measurement invariance was tested via a multiple group confirmatory factor analysis. The assessment of invariance included the three levels of configural, metric, and scalar invariance, and the latent means of the values between both samples were compared. Results of this study show that the measurements are invariant at the three levels (configural, metric, and scalar), but there are latent mean differences between the values across the surveys. These differences may be partly explained by age and level of education differences between the two samples. Based on these findings we conclude that the methods of measurement are essentially invariant.     

Applications of calendar instruments in social surveys: a review

Retrospective reports in survey interviews and questionnaires are subject to many types of recall error, which affect completeness, consistency, and dating accuracy. Concerns about this problem have led to the development of so-called calendar instruments, or timeline techniques. These aided recall procedures have been designed to help respondents gain better access to long-term memory by providing a graphical time frame in which life history information can be represented. In order to obtain more insights into the potential benefits of calendar methodology, this paper presents a review of the application of calendar instruments, their design characteristics and effects on data quality. Calendar techniques are currently used in a variety of fields, including life course research, epidemiology and family planning studies. Despite the growing interest in these new methods, their application often lacks sufficient theoretical foundation and little attention has been paid to their effectiveness. Several recent studies however, have demonstrated that in comparison to more traditional survey methods, calendar techniques can improve some aspects of data quality. While calendar instruments have been shown to be potentially beneficial to retrospective data quality, there is an apparent need for methodological research that generates more systematic knowledge about their application in social surveys.     

Organizational innovation: The challenge of measuring non-technical innovation in large-scale surveys

Although the implementation of innovative organizational concepts is considered to be highly important for a company's competitiveness, so far there has been little research on possible approaches to measure and monitor organizational innovations in large-scale surveys. Based on an item-oriented typology of organizational innovations which serves as the precondition for a common understanding, we describe and compare how organizational innovations have been measured through existing surveys in Europe. Using a large-scale survey comprising data of 1450 German manufacturing companies, we show how these different approaches lead to significantly different results regarding the organizational innovativeness of companies within one and the same sample. We derive four implications for the future measurement and monitoring of organizational innovations. Our findings contribute to the further development of an adequate methodology for an organizational innovation monitoring system.     

Cognitive interviewing and factor-analytic techniques: a mixed method approach to validity of survey items measuring national identity

This paper explores the possibilities of method triangulation between two methodological approaches for assessing the validity performance of survey items: cognitive interviewing and factor analytic techniques. Although their means of approaching validity differ, both methods attempt to prove whether a measure corresponds to a theoretical (latent) concept (e.g. patriotism vs. nationalism), thus both are concerned with the question, whether an indicator measures what it is supposed to measure. Based on two representative samples for Austria [data gathered within the framework of the International Social Survey Program (ISSP) on National Identity in 1995 and 2003] and 18 cognitive interviews conducted between 2003 and 2005, the paper shows the considerable advantages of using a multi-method approach for ensuring the quality of survey items. On the one hand, we apply exploratory and confirmatory factor analysis in order to identify poorly performing indicators with regard to validity and reliability. On the other hand, the analysis of the cognitive interviews reveals the substantial sources of response error. Results show that to a large extent, respondents do not understand the items that have been defined to measure national identification and related concepts in Austria the way intended by the drafting group of this ISSP Module, a fact that has considerable implications on the scales’ predictive power.     

Modeling Clustered Ordered Categorical Data: A Survey

This article surveys various strategies for modeling ordered categorical (ordinal) response variables when the data have some type of clustering, extending a similar survey for binary data by Pendergast, Gange, Newton, Lindstrom, Palta & Fisher (1996). An important special case is when repeated measurement occurs at various occasions for each subject, such as in longitudinal studies. A much greater variety of models and fitting methods are available than when a similar survey for repeated ordinal response data was prepared a decade ago (Agresti, 1989). The primary emphasis of the review is on two classes of models, marginal models for which effects are averaged over all clusters at particular levels of predictors, and cluster-specific models for which effects apply at the cluster level. We present the two types of models in the ordinal context, review the literature for each, and discuss connections between them. Then, we summarize some alternative modeling approaches and ways of estimating parameters, including a Bayesian approach. We also discuss applications and areas likely to be popular for future research, such as ways of handling missing data and ways of modeling agreement and evaluating the accuracy of diagnostic tests. Finally, we review the current availability of software for using the methods discussed in this article.     

Scoring rules and survey density forecasts

This article provides a practical evaluation of some leading density forecast scoring rules in the context of forecast surveys. We analyse the density forecasts of UK inflation obtained from the Bank of England’s Survey of External Forecasters, considering both the survey average forecasts published in the Bank’s quarterly Inflation Report, and the individual survey responses recently made available to researchers by the Bank. The density forecasts are collected in histogram format, and the ranked probability score (RPS) is shown to have clear advantages over other scoring rules. Missing observations are a feature of forecast surveys, and we introduce an adjustment to the RPS, based on the Yates decomposition, to improve its comparative measurement of forecaster performance in the face of differential non-response. The new measure, denoted RPS*, is recommended to analysts of forecast surveys.     

Print versus electronic surveys: a comparison of two data collection methodologies

This paper compares the responses of consumers who submitted answers to a survey instrument focusing on Internet purchasing patterns both electronically and using traditional paper response methods. We present the results of a controlled experiment within a larger data collection effort. The same survey instrument was completed by 416 Internet customers of a major office supplies company, with approximately 60% receiving the survey in paper form and 40% receiving the electronic version. In order to evaluate the efficacy of electronic surveys relative to traditional, printed surveys we conduct two levels of analysis. On a macro-level, we compare the two groups for similarity in terms of fairly aggregate, coarse data characteristics such as response rates, proportion of missing data, scale means and inter-item reliability. On a more fine-grained, micro-level, we compare the two groups for aspects of data integrity such as the presence of data runs and measurement errors. This deeper, finer-grained analysis allows an examination of the potential benefits and flaws of electronic data collection.Our findings suggest that electronic surveys are generally comparable to print surveys in most respects, but that there are a few key advantages and challenges that researchers should evaluate. Notably, our sample indicates that electronic surveys have fewer missing responses and can be coded/presented in a more flexible manner (namely, contingent coding with different respondents receiving different questions depending on the response to earlier questions) that offers researchers new capabilities.     

Improving retrospective life course data by combining modularized self-reports and event history calendars: experiences from a large scale survey

Event history calendars (EHC) have proven to be a powerful tool for collecting retrospective autobiographical life course data. One problem is that they are only standardized to a limited extent. This restricts their applicability in large-scale surveys. However, in such surveys, a modularized retrospective CATI design can be combined with an EHC. This data revision module is directly integrated into the interview and used as a data revision module, allowing insights from cognitive psychology to be applied. The data revision module stimulates the respondent’s memory retrieval by detecting both temporal inconsistencies, such as gaps, and overlapping or parallel events. This approach was implemented in the IAB-ALWA study (Working and Learning in a Changing World), a large-scale representative telephone survey involving 10,000 respondents. By comparing the uncorrected data with the final data after revision, we can investigate to what extent the application of this data revision module improves data quality or, more precisely, time consistency and dating accuracy of individual reports.     

Properties of Census Dual System Population Size Estimators

We study parametric and non‐parametric approaches for assessing the accuracy and coverage of a population census based on dual system surveys. The two parametric approaches being considered are post‐stratification and logistic regression, which have been or will be implemented for the US Census dual system surveys. We show that the parametric model‐based approaches are generally biased unless the model is correctly specified. We then study a local post‐stratification approach based on a non‐parametric kernel estimate of the Census enumeration functions. We illustrate that the non‐parametric approach avoids the risk of model mis‐specification and is consistent under relatively weak conditions. The performances of these estimators are evaluated numerically via simulation studies and an empirical analysis based on the 2000 US Census post‐enumeration survey data.     

National minorities and their representation in social surveys: which practices make a difference?

This paper presents a systematic study of survey mechanisms that produce or reduce minority bias in social surveys. It extends the work of Lipps et al. (2011) who have demonstrated that, in the Swiss context, the more an ethno-national minority community differs, socio-culturally and socio-economically, from the national majority, the less it is likely to be represented in its proper proportion in the major national surveys. Minority bias furthermore has a vertical dimension: socio-economic bias against individuals from the most deprived backgrounds becomes extreme within ethno-national minority communities. Using data from the Swiss Labour Force Survey, the Swiss Household Panel, and the Swiss sample of the European Social Survey, in the present work we empirically assess the impact of three types of survey practices on minority bias: (1) strategies to increase overall response rates of the whole population indiscriminately from its minority status, (2) the use of pre- and post-stratification measures that take into account the overall share of foreigners in the national population, and (3) the conduct of interviews in a wider range of languages, in order to facilitate survey response among certain (linguistic) minorities. Our findings show that efforts to increase overall response rates can, paradoxically, create even more minority bias. On the other hand, they suggest that a combination of stratified sampling and a wider range of survey languages can have a positive effect in reducing survey bias, both between and within national categories. We conclude that measures that take into account and adapt to the social and cultural heterogeneity of surveyed populations do make a difference, whereas additional efforts that only replicate existing routine practices can be counter-productive.