Print versus electronic surveys: a comparison of two data collection methodologies

This paper compares the responses of consumers who submitted answers to a survey instrument focusing on Internet purchasing patterns both electronically and using traditional paper response methods. We present the results of a controlled experiment within a larger data collection effort. The same survey instrument was completed by 416 Internet customers of a major office supplies company, with approximately 60% receiving the survey in paper form and 40% receiving the electronic version. In order to evaluate the efficacy of electronic surveys relative to traditional, printed surveys we conduct two levels of analysis. On a macro-level, we compare the two groups for similarity in terms of fairly aggregate, coarse data characteristics such as response rates, proportion of missing data, scale means and inter-item reliability. On a more fine-grained, micro-level, we compare the two groups for aspects of data integrity such as the presence of data runs and measurement errors. This deeper, finer-grained analysis allows an examination of the potential benefits and flaws of electronic data collection.Our findings suggest that electronic surveys are generally comparable to print surveys in most respects, but that there are a few key advantages and challenges that researchers should evaluate. Notably, our sample indicates that electronic surveys have fewer missing responses and can be coded/presented in a more flexible manner (namely, contingent coding with different respondents receiving different questions depending on the response to earlier questions) that offers researchers new capabilities.     

Determinants of social desirability bias in sensitive surveys: a literature review

Survey questions asking about taboo topics such as sexual activities, illegal behaviour such as social fraud, or unsocial attitudes such as racism, often generate inaccurate survey estimates which are distorted by social desirability bias. Due to self-presentation concerns, survey respondents underreport socially undesirable activities and overreport socially desirable ones. This article reviews theoretical explanations of socially motivated misreporting in sensitive surveys and provides an overview of the empirical evidence on the effectiveness of specific survey methods designed to encourage the respondents to answer more honestly. Besides psychological aspects, like a stable need for social approval and the preference for not getting involved into embarrassing social interactions, aspects of the survey design, the interviewer’s characteristics and the survey situation determine the occurrence and the degree of social desirability bias. The review shows that survey designers could generate more valid data by selecting appropriate data collection strategies that reduce respondents’ discomfort when answering to a sensitive question.     

An assessment of equivalence between Internet and paper-based surveys: evidence from collectivistic cultures

Little research exists that addresses the equivalence in collectivistic cultures of paper- versus Internet-based surveys. This study addressed this gap and examined the measurement equivalence of individual innovativeness scales between Internet surveys and paper-based surveys within a collectivistic culture (with China serving as our example). The study analyzed and compared survey data from both paper and web-based surveys using confirmatory factor analysis. The assessment of invariance included the levels of configural, metric, scalar, and covariance invariance. The means and variance of latent variables were also compared. The results show that measurements are invariant at the two levels (configural and metric), and the covariances between latent variables are also equivalent, but the mean and variance differences of latent variables are apparent. The results indicate that when conducting research in collectivistic cultures and collecting data from distinct survey modes, researchers should concern themselves with the potential of extreme response patterns and the inclination of social desirability responding, as well as considering the measurement invariance across survey modes.     

Conducting Online Surveys

The World Wide Web (WWW) is increasingly being used as a tool and platform for survey research. Two types of electronic or online surveys available for data collection are the email and Web based survey, and they constitute the focus of this paper. We address a multitude of issues researchers should consider before and during the use of this method of data collection: advantages and liabilities with this form of survey research, sampling problems, questionnaire design considerations, suggestions in approaching potential respondents, response rates and aspects of data processing. Where relevant, the methodological issues involved are illustrated with examples from our own research practice. This methods review shows that most challenges are resolved when taking into account the principles that guide the conduct of conventional surveys.     

Evaluating,Comparing, Monitoring,and Improving Representativeness of Survey Response Through R‐Indicators and Partial R‐Indicators

Non‐response is a common source of error in many surveys. Because surveys often are costly instruments, quality‐cost trade‐offs play a continuing role in the design and analysis of surveys. The advances of telephone, computers, and Internet all had and still have considerable impact on the design of surveys. Recently, a strong focus on methods for survey data collection monitoring and tailoring has emerged as a new paradigm to efficiently reduce non‐response error. Paradata and adaptive survey designs are key words in these new developments. Prerequisites to evaluating, comparing, monitoring, and improving quality of survey response are a conceptual framework for representative survey response, indicators to measure deviations thereof, and indicators to identify subpopulations that need increased effort. In this paper, we present an overview of representativeness indicators or R‐indicators that are fit for these purposes. We give several examples and provide guidelines for their use in practice.     

Social Desirability Bias in Survey Research on Sustainable Development in Small Firms: an Exploratory Analysis of Survey Mode Effect

Survey methodology has become an increasingly popular weapon of choice in conducting research on the topic of environmental sustainability in the small firm context. However, conventional application of survey research in many empirical studies tends to ignore the inherent issues associated with the mode of administration of survey questionnaires. One of these issues is social desirability bias (SDB) arising from survey modes such as face to face, the assisted type of survey questionnaire administration and mailed or purely self‐administered survey. This study is an analysis of survey data generated by two modes: purely self‐administered and survey enumerator assisted. The paper argues that the presence of an enumerator in the workplace or actual site of the business to administer the survey will motivate the respondents to provide more accurate information about the firms' environmental sustainability knowledge and practices. The article offers suggestions to address SDB in carrying out surveys about environmental sustainability in small firms. Copyright © 2011 John Wiley & Sons, Ltd and ERP Environment.     

Assessing the Effect of Data Collection Mode on Measurement

This paper discusses the challenges in assessing the effects of mixed modes on measurement. We briefly review theories about why different modes of data collection can lead to differences in survey responses and statistical methods typically used to assess mode effects. We then discuss the challenges, including which mode effects are identified, how to test for mode effects, and whether these would affect substantive conclusions. The issues raised are illustrated with examples from the European Social Survey, which is conducting a programme of experimental research to inform decisions about whether to use mixed modes of data collection. The paper concludes with general implications for mixed modes research.     

Selecting Adaptive Survey Design Strata with Partial R‐indicators

Recent survey literature shows an increasing interest in survey designs that adapt data collection to characteristics of the survey target population. Given a specified quality objective function, the designs attempt to find an optimal balance between quality and costs. Finding the optimal balance may not be straightforward as corresponding optimisation problems are often highly non‐linear and non‐convex. In this paper, we discuss how to choose strata in such designs and how to allocate these strata in a sequential design with two phases. We use partial R‐indicators to build profiles of the data units where more or less attention is required in the data collection. In allocating cases, we look at two extremes: surveys that are run only once, or infrequent, and surveys that are run continuously. We demonstrate the impact of the sample size in a simulation study and provide an application to a real survey, the Dutch Crime Victimisation Survey.     

Cognitive interviewing as tool for enhancing the accuracy of the interpretation of quantitative findings

This paper contrasts findings from a quantitative survey with those from a cognitive interviewing follow-up investigation on a subset of the same respondents. The data were gathered as part of a larger study to explore measurement error across three modes of data collection, but this paper focuses on the question format experiments rather than the mode effects part of the larger study. Three examples are presented which demonstrate how cognitive interviewing can cast new light on quantitative results by increasing the accuracy of the inferences made. These include instances where: (1) quantitative indicators of poor respondent behaviour (e.g., acquiescence bias on agree/disagree questions) are over-estimates, (2) similar quantitative response distributions across satisfaction and behavioural questions (from a fully-labelled versus end-labelled experiment) imply similar respondent satisficing behaviour, but cognitive interviews show that different response processes are at work and (3) unlikely quantitative findings (from an experiment comparing 3 vs. 7 or 8 response options) could easily be dismissed as due to chance but were instead the result of unforeseen respondent difficulties. The paper concludes with a discussion of the value of using a cognitive interviewing follow-up study as a tool in the interpretation of ambiguous quantitative findings.     

Certifying interviewers: the role of testing interviewers to improve data quality

The complexity of conducting personal interviews often requires both classroom training prior to fielding a survey and the certification of interviewers in the field. In an attempt to create a tool that would enable data collection agencies to tailor the training process and more effectively link the classroom to the field, the Census Bureau administered a certification exam to the interviewers of a new, more conversational style of interviewing in the redesign of the Survey of Income and Program Participation-Event History Calendar (SIPP-EHC). This research applied multilevel modeling to the 2010 and 2011 SIPP-EHC field test data to observe the effects of the variation in certification test score on measures of interviewer productivity—interview length and person non-response. From 2010 to 2011, short answer response questions replaced half of the multiple-choice response questions, finding short answer responses are better predictors of both interview length and person non-response than multiple-choice responses. These changes in the exam demonstrated a significant reduction in the interviewer effect seen on both outcomes.     

Applications of calendar instruments in social surveys: a review

Retrospective reports in survey interviews and questionnaires are subject to many types of recall error, which affect completeness, consistency, and dating accuracy. Concerns about this problem have led to the development of so-called calendar instruments, or timeline techniques. These aided recall procedures have been designed to help respondents gain better access to long-term memory by providing a graphical time frame in which life history information can be represented. In order to obtain more insights into the potential benefits of calendar methodology, this paper presents a review of the application of calendar instruments, their design characteristics and effects on data quality. Calendar techniques are currently used in a variety of fields, including life course research, epidemiology and family planning studies. Despite the growing interest in these new methods, their application often lacks sufficient theoretical foundation and little attention has been paid to their effectiveness. Several recent studies however, have demonstrated that in comparison to more traditional survey methods, calendar techniques can improve some aspects of data quality. While calendar instruments have been shown to be potentially beneficial to retrospective data quality, there is an apparent need for methodological research that generates more systematic knowledge about their application in social surveys.     

Multiple Visits and Data Quality in Household Surveys

In order to increase data quality some household surveys visit the respondent households several times to estimate one measure of consumption. For example, in Ghanaian Living Standards Measurement surveys, households are visited up to 10 times over a period of 1 month. I find strong evidence for conditioning effects as a result of this approach: In the Ghanaian data the estimated level of consumption is a function of the number of prior visits, with consumption being highest in the earlier survey visits. Telescoping (perceiving events as being more recent than they are) or seasonality (first‐of‐the‐month effects) cannot explain the observed pattern. To study whether earlier or later survey visits are of higher quality, I employ a strategy based on Benford's law. Results suggest that the consumption data from earlier survey visits are of higher quality than data from later visits. The findings have implications for the value of additional visits in household surveys, and also shed light on possible measurement problems in high‐frequency panels. They add to a recent literature on measurement errors in consumption surveys (Beegle et al., 2012 , Gibson et al., 2015 ), and complement findings by Zwane et al. ( 2011 ) regarding the effect of surveys on subsequent behaviour.     

Statistics by Gender: Measures to Reduce Gender Bias in Agricultural Surveys

Statistics by gender has been the concern of policy makers in the recent past years. The demand on data disaggregated by gender has led the survey statistician to collect data and tabulate statistics by gender. In this paper, some measures will be suggested to avoid and reduce gender bias for data collection and tabulation in agricultural surveys.     

Determining the impact of age and LOS on job satisfaction: a case study of Pakistan automotive industry

This research was conducted to ascertain the impact of age and length of service (LOS) on job satisfaction in engineers of Pakistan public sector. Field survey was conducted using job satisfaction survey (JSS) questionnaire having closed-ended questions. Multistage sampling was conducted using a combination of cluster sampling, stratified sampling and random sampling techniques. Power and Precision software was used to determine the sample size. JSS questionnaires were administered amongst 225 electrical and mechanical engineers from five public sector organizations. 158 usable questionnaires were received and data were analyzed in SPSS. Statistical analyses showed existence of an open mouth U-shaped relationship between LOS/age and job satisfaction. It was found that age moderates relationship between LOS and job satisfaction. Non-responsiveness of senior engineers led to one of the limitations of this study. Results of this study can be used for policy-making decisions.     

Assessing the economic knowledge and economic opinions of adults

This study investigates public knowledge of basic economics and public opinion on economic issues. The primary data sources are five national surveys, administered from 1992 to 1999, which contain a rich set of questions to conduct multiple tests and comparisons of the factors that affect economic knowledge and public opinion. As a whole, the results offer significantly stronger evidence of factors that influence knowledge and opinion than is possible from a study of a particular sample of adults using a single set of survey questions.The analysis proceeds in two ways following methods that were originally used with one of the five data sets, a 1992 survey of adults (Walstad, 1997). First, a regression model is specified and estimated with each data set to identify how personal characteristics, general education, course work in economics, income, and political party affect economic knowledge. Second, probit analysis is used to evaluate the effect of economic knowledge on public opinion on selected economic issues after controlling for the above variables. The results from the 1992 data serve as the baseline for comparing the findings across the other surveys.     

Selection Bias in Web Surveys

At first sight, web surveys seem to be an interesting and attractive means of data collection. They provide simple, cheap, and fast access to a large group of potential respondents. However, web surveys are not without methodological problems. Specific groups in the populations are under-represented because they have less access to Internet. Furthermore, recruitment of respondents is often based on self-selection. Both under-coverage and self-selection may lead to biased estimates. This paper describes these methodological problems. It also explores the effect of various correction techniques (adjustment weighting and use of reference surveys). This all leads to the question whether properly design web surveys can be used for data collection. The paper attempts to answer this question. It concludes that under-coverage problems may solve itself in the future, but that self-selection leads to unreliable survey outcomes.     

How should Mass attendance be measured? An Italian case study

The validity of measures of Mass attendance has been a matter of controversy for some time in Italy, just as it has in other countries. This study will show that the different survey methods which can be used each have distinct advantages and disadvantages. We will compare the questionnaire and daily diary as measuring devices and try to highlight their main qualities and failings. More specifically, although the large number of up-to-date questionnaire-based surveys carry the benefit of offering a ??richer?? range of information regarding the interviewee??s religious behaviour, they provide data which is excessively approximate on a systematic basis. In comparison, time use studies are less common and much more costly, partly because they usually use a daily diary to record the activities carried out. Although the data they produce is ??scarcer?? in terms of information, it is also much more reliable. Indeed, estimates obtained from comparisons between the two tools highlight significant differences. In addition, synchronic and diachronic comparisons cannot be carried out due to the fact that the distortions introduced by the use of questionnaires are extremely variable over time and depending on the social segments studied (young people/adults, men/women etc.). The conclusion we will draw is that the daily diary is a far more suitable tool for surveying Mass attendance than the questionnaire because of these distortions. Finally, we will show that this conclusion does not depend on the specific survey methods adopted, as comparisons with multination surveys seem to confirm that, at least with regard to Italy, distortion resulting from use of a questionnaire is a general phenomenon.     

The long and short of asking questions about income: a comparison using data from Hungary

A lot of research on income mobility and income inequality is based on survey questions about income. Various question formats are being used. Researchers seem to assume that the actual format used delivers the best estimate of the “true” income. However, surprisingly little empirical support is available for this claim. We implemented an experimental design using the short and long versions of income questions in a Hungarian survey. Results show an overall positive difference between the long and short version. The differences are related to the income components (wages and salaries, transfers, and assets), and respondent characteristics, controlling for the effect of the order of the two versions of income questions. Based on the results, we provide some recommendations for implementing income questions in surveys.     

Two new models for survey sampling with sensitive characteristic: design and analysis

Sensitive topics or highly personal questions are often being asked in medical, psychological and sociological surveys. This paper proposes two new models (namely, the triangular and crosswise models) for survey sampling with the sensitive characteristics. We derive the maximum likelihood estimates (MLEs) and large-sample confidence intervals for the proportion of persons with sensitive characteristic. The modified MLEs and their asymptotic properties are developed. Under certain optimality criteria, the designs for the cooperative parameter are provided and the sample size formulas are given. We compare the efficiency of the two models based on the variance criterion. The proposed models have four advantages: neither model requires randomizing device, the models are easy to be implemented for both interviewer and interviewee, the interviewee does not face any sensitive questions, and both models can be applied to both face-to-face personal interviews and mail questionnaires.     

Institutional entrepreneurship and permaculture: A practice theory perspective

Permaculture is a growing but little researched phenomenon emphasising care for the environment, equity, fair treatment of people and working with—and not against—nature. It thus represents a potential alternative to business as usual, capable of addressing fundamental challenges posed by human-made climate change. The paper examines a previously ignored site of entrepreneurship by taking a practice perspective, exploring connections between the practice and growth of permaculture and institutional entrepreneurship. It assesses practice-related and institutional factors affecting the start-up and operation of permaculture enterprises in the United Kingdom. The study maps and surveys UK Permaculture Association members who have started up their own business and reports on qualitative data from personal interviews with twenty of them. Data analysis employs NVivo software and involves thematic analysis pertaining to the practice, institutional biographies and institutional portfolios of permaculture entrepreneurs. The findings show the importance of permaculture activists' institutional biographies and institutional portfolios to the start-up and operation of permaculture enterprises and for shaping permaculture-related practice. The contribution of the paper lies in how it balances attention to individual agency with subfield-specific, organisational field and macrosocial factors in understanding ‘beyond profit’ entrepreneurship.