Conducting Online Surveys

The World Wide Web (WWW) is increasingly being used as a tool and platform for survey research. Two types of electronic or online surveys available for data collection are the email and Web based survey, and they constitute the focus of this paper. We address a multitude of issues researchers should consider before and during the use of this method of data collection: advantages and liabilities with this form of survey research, sampling problems, questionnaire design considerations, suggestions in approaching potential respondents, response rates and aspects of data processing. Where relevant, the methodological issues involved are illustrated with examples from our own research practice. This methods review shows that most challenges are resolved when taking into account the principles that guide the conduct of conventional surveys.     

Mixture of Truthful–Untruthful Responses in Public Surveys

When sensitive issues are surveyed, collecting truthful data and obtaining reliable estimates of population parameters is a persistent problem in many fields of applied research mostly in sociological, economic, demographic, ecological and medical studies. In this context, and moving from the so‐called negative survey, we consider the problem of estimating the proportion of population units belonging to the categories of a sensitive variable when collected data are affected by measurement errors produced by untruthful responses. An extension of the negative survey approach is proposed herein in order to allow respondents to release a true response. The proposal rests on modelling the released data with a mixture of truthful and untruthful responses that allows researchers to obtain an estimate of the proportions as well as the probability of receiving the true response by implementing the EM‐algorithm. We describe the estimation procedure and carry out a simulation study to assess the performance of the EM estimates vis‐à‐vis certain benchmark values and the estimates obtained under the traditional data‐collection approach based on direct questioning that ignores the presence of misreporting due to untruthful responding. Simulation findings provide evidence on the accuracy of the estimates and permit us to appreciate the improvements that our approach can produce in public surveys, particularly in election opinion polls, when the hidden vote problem is present.     

Evaluating,Comparing, Monitoring,and Improving Representativeness of Survey Response Through R‐Indicators and Partial R‐Indicators

Non‐response is a common source of error in many surveys. Because surveys often are costly instruments, quality‐cost trade‐offs play a continuing role in the design and analysis of surveys. The advances of telephone, computers, and Internet all had and still have considerable impact on the design of surveys. Recently, a strong focus on methods for survey data collection monitoring and tailoring has emerged as a new paradigm to efficiently reduce non‐response error. Paradata and adaptive survey designs are key words in these new developments. Prerequisites to evaluating, comparing, monitoring, and improving quality of survey response are a conceptual framework for representative survey response, indicators to measure deviations thereof, and indicators to identify subpopulations that need increased effort. In this paper, we present an overview of representativeness indicators or R‐indicators that are fit for these purposes. We give several examples and provide guidelines for their use in practice.     

Improving the response rate and quality in Web-based surveys through the personalization and frequency of reminder mailings

Internet shares some characteristics of survey making with traditional media, especially postal mail. However, there are considerable differences that justify a different focus on administration and make existing knowledge of the traditional media not directly applicable to Internet. Research is therefore necessary to discover how Web-based surveys operate under different conditioning factors, so that general behavioral patterns can be established in order to improve the administration and results of such surveys. This study thus centers on two of the parameters that can influence responses to Web-based surveys, which are personalization and the frequency of reminder mailings distributed among the sample population. The results obtained show a positive influence of personalized e-mail messages on response rate and the need to use a lower frequency for studies aiming at increasing the response rate in the shortest possible time; and longer frequency (and personalized) when the aim is to have the respondents complete the full questionnaire.     

An assessment of equivalence between Internet and paper-based surveys: evidence from collectivistic cultures

Little research exists that addresses the equivalence in collectivistic cultures of paper- versus Internet-based surveys. This study addressed this gap and examined the measurement equivalence of individual innovativeness scales between Internet surveys and paper-based surveys within a collectivistic culture (with China serving as our example). The study analyzed and compared survey data from both paper and web-based surveys using confirmatory factor analysis. The assessment of invariance included the levels of configural, metric, scalar, and covariance invariance. The means and variance of latent variables were also compared. The results show that measurements are invariant at the two levels (configural and metric), and the covariances between latent variables are also equivalent, but the mean and variance differences of latent variables are apparent. The results indicate that when conducting research in collectivistic cultures and collecting data from distinct survey modes, researchers should concern themselves with the potential of extreme response patterns and the inclination of social desirability responding, as well as considering the measurement invariance across survey modes.     

New data collection modes for surveys: a comparative analysis of the influence of survey mode on question-wording effects

New opportunities have recently emerged in survey practice, coupled with a need to make changes, and alternative survey data collection modes such as those based on new technologies (for example, the Web and mobile phones) have become a focus of interest. Studies have considered the biases due to data collection modes and to the wording of questions in questionnaires, but they have rarely dealt with the interaction between the two phenomena. This paper presents the results of a study on the interactions between some of the best-known question-wording effects and three data collection modes: face-to-face, Web-based and SMS-based (Short Text Messaging). The results have highlighted some interesting characteristics of the various modes—innovative ones in particular—and have confirmed the existence of potential interactions between data collection modes and question-wording effects. The findings may have significant implications for the study and practice of surveys and entail that, when surveys are designed, account is taken of specific factors associated with the method used to word the questions in questionnaires.     

Attitudes Toward Surveys, Attitude Accessibility and the Effect on Respondents’ Susceptibility to Nonresponse

This paper analyzes whether respondents’ attitudes toward surveys explain their susceptibility to item nonresponse. In contrast to previous studies, the decision to refuse to provide income information, not to answer other questions and the probability of ‘don’t know’ responses is tested separately. Furthermore, the interviewers’ overall judgments of response willingness was included as well. Respondents with a positive and cognitively accessible attitude toward surveys were expected to adopt a cooperative orientation and were thus deemed more likely to answer difficult as well as sensitive questions. Attitudes were measured with a 16-item instrument and the response latencies were used as an indicator for attitude accessibility. We found that respondents with more favorable evaluations of surveys had lower values on all kinds of nonresponse indicators. Except for the strong effect on the prevalence of ‘don’t knows’, survey attitudes were increasingly more predictive for all other aspects of nonresponse when these attitude answers were faster and thus cognitively more accessible. This accessibility, and thus how relevant survey attitudes are for nonresponse, was found to increase with the subjects’ exposure to surveys in the past.     

Experimental comparison of Web, electronic and mail survey technologies in operations management

With the growing acceptance of the Web (Internet) and electronic mail, it is no surprise that researchers are using an increasingly diverse set of survey technologies to gather data from managers. However, the effectiveness of these electronic technologies has not been rigorously assessed, especially for gathering data from establishment-level surveys (i.e. firm- or plant-level). To that end, a stratified sample of large and small, service and manufacturing firms was constructed, followed by random assignment to one of four survey technologies: mail, fax, PC disk-by-mail and Web-page survey (combined with e-mail notification). For each treatment, managers are queried about their use of forecasting characteristics, yielding a sample of 118 firms. Unfortunately, only a low percentage (34%) of firms and managers assigned to the Web technology treatment both reported access to e-mail and were willing provide their e-mail addresses; they tended to be large firms and from the service sector. Moreover, those that did offer e-mail addresses were only about half as likely to respond to the Web-based survey as those targeted by other survey technologies. However, Web, fax and disk-by-mail technologies yielded higher item completion rates than mail. Limited statistical evidence indicated that respondents using computer-based survey technologies (i.e. Web or disk-by-mail) generally reported forecasting characteristics that are associated with firms exhibiting best practices. Thus, a multi-technology survey approach using the Web and fax can yield a strong combination of benefits over a traditional mail survey.     

Selection Bias in Web Surveys

At first sight, web surveys seem to be an interesting and attractive means of data collection. They provide simple, cheap, and fast access to a large group of potential respondents. However, web surveys are not without methodological problems. Specific groups in the populations are under-represented because they have less access to Internet. Furthermore, recruitment of respondents is often based on self-selection. Both under-coverage and self-selection may lead to biased estimates. This paper describes these methodological problems. It also explores the effect of various correction techniques (adjustment weighting and use of reference surveys). This all leads to the question whether properly design web surveys can be used for data collection. The paper attempts to answer this question. It concludes that under-coverage problems may solve itself in the future, but that self-selection leads to unreliable survey outcomes.     

A comparison of open-ended and closed questions in the prediction of mental health

The statistical benefit of adding open-ended questions to closed questions was evaluated in a survey of 643 participants. The construct of coping was chosen as the measurement domain. Open and closed questions were used to predict mental health a year later. Verbatim responses to open questions were reliably coded (ICC = 0.92), but they did not increase the statistical prediction of measures of mental health beyond the contribution of closed questions. Open-ended questions provided more in-depth information than closed questions, but at the cost of more missing data and less degrees of freedom. The benefit of using open-ended in addition to standard closed questions was thus practically nil, hence questioning the use of qualitative information gathering in surveys for the purpose of statistical prediction.     

Selecting Adaptive Survey Design Strata with Partial R‐indicators

Recent survey literature shows an increasing interest in survey designs that adapt data collection to characteristics of the survey target population. Given a specified quality objective function, the designs attempt to find an optimal balance between quality and costs. Finding the optimal balance may not be straightforward as corresponding optimisation problems are often highly non‐linear and non‐convex. In this paper, we discuss how to choose strata in such designs and how to allocate these strata in a sequential design with two phases. We use partial R‐indicators to build profiles of the data units where more or less attention is required in the data collection. In allocating cases, we look at two extremes: surveys that are run only once, or infrequent, and surveys that are run continuously. We demonstrate the impact of the sample size in a simulation study and provide an application to a real survey, the Dutch Crime Victimisation Survey.     

Assessing nonresponse bias in activist surveys

How confident can we be that the comparatively low response rates associated with mail surveys of groups of political activists, such as participants of a demonstration, does not hide a substantial nonresponse bias? The paper compares the results of a face-to-face survey of 2003 anti-Iraq war demonstrators in Glasgow, achieving a near perfect response rate, with the data derived from a mail survey handed out to demonstrators eliciting valid responses from 37% of marchers. The comparison shows that better educated, older, female demonstrators were more likely to return the mail questionnaire. Also demonstrators having born a higher ‘cost’ of travelling to the demonstration are more likely to respond. There was no evidence that political interest or political orientation played an important role. However, those who had taken part in demonstrations very frequently in recent years were less likely to return the mail questionnaire. While these results provide some reassurance that even with response rates below 40%, no substantive political bias is present, researchers undertaking surveys of activists should be alerted to the need to address possible nonresponse biases in a systematic way.     

Mail survey abroad with an alternative web survey

The large decline in cooperation in surveys experienced in the first decade of the twenty first century has placed representative surveys in a veritable quagmire, although numerous researchers have shown that the decline in the total nonresponse has been much higher in administered surveys (face-to-face and telephone) than in self-administered (mail and web surveys). The move from the “traditional” mail survey to the internet survey has brought important changes, but, at the same time, new challenges have arisen related to the lower coverage and the reduced visibility, the need for certain skills to fill out a questionnaire, changes in response rates, etc. This is a self-administered survey that informs the addressees by mail that they have been selected to participate in a research survey to which they can respond either by using the paper questionnaire (in the letter there is a prepaid envelope), by means of a web questionnaire (accessing a link and using a password), or by telephone (the respondent provided a contact time telephone number via a prepaid envelope or by calling an answering machine). The respondents themselves, therefore, choose the mode of response. An analysis of the responses achieved by each modality shows that the majority of the questionnaires were received by ordinary mail (83.6 %), 359 by internet (14.4 %), and 51 were answered by telephone (2.0 %).     

DATA GAPS,DATA INCOMPARABILITY,AND DATA IMPUTATION: A REVIEW OF POVERTY MEASUREMENT METHODS FOR DATA‐SCARCE ENVIRONMENTS

Questions that often come up in contexts where household consumption data are unavailable or missing include: what are the best existing methods to obtain poverty estimates at a single snapshot in time? and over time? and what are the best available methods to study poverty dynamics? A variety of different techniques have been developed to tackle these questions, but unfortunately, they are presented in different forms and lack unified terminology. We offer a review of poverty imputation methods that address contexts ranging from completely missing and partially missing consumption data in cross‐sectional household surveys, to missing panel household data. We present the various existing methods under a common framework, with pedagogical discussion on their intuition. Empirical illustrations are provided using several rounds of household survey data from Vietnam. Furthermore, we also offer a practical guide with detailed instructions on computer programs that can be used to implement the reviewed techniques.     

Accuracy of self-reports to the burdensome question: survey response and nonresponse error trade-offs

Survey self-reports to questions involving respondent burden are examined for: (1) overall accuracy, (2) direction of bias, and (3) influence on relationships. Self-reports by respondents to questions on an area of household finances are compared to error-free supplier records. The evidence shows self-reports in this case to be accurate at the overall, or aggregate, level, but inaccurate at the individual, or disaggregate, level. The trade-offs inherent in the nonresponse occurring as a consequence of attempting to acquire verified data versus response bias resulting from a reliance on self-reports from surveys are discussed. It is suggested that attempts by researchers to acquire error-free, verified information in surveys will lead in most cases to unacceptably high levels of item nonresponse. An alternative, algebraic procedure for adjusting for disaggregate level response bias in self-reports on the basis of a subsample of verified responses is provided.     

Alternative Indicators for the Risk of Non‐response Bias: A Simulation Study

The growth of non‐response rates for social science surveys has led to increased concern about the risk of non‐response bias. Unfortunately, the non‐response rate is a poor indicator of when non‐response bias is likely to occur. We consider in this paper a set of alternative indicators. A large‐scale simulation study is used to explore how each of these indicators performs in a variety of circumstances. Although, as expected, none of the indicators fully depict the impact of non‐response in survey estimates, we discuss how they can be used when creating a plausible account of the risks for non‐response bias for a survey. We also describe an interesting characteristic of the fraction of missing information that may be helpful in diagnosing not‐missing‐at‐random mechanisms in certain situations.     

Development in Information and Communications Technology (ICT) Statistics at the Australian Bureau of Statistics

This paper outlines the ICT statistical developments that the ABS has undertaken over recent years. These developments have been on both the supply side (ICT industry) and the demand side (use of IT by sector). The content of ABS ICT surveys has changed significantly over the last few years and are expected to develop further in line with industry changes and emerging policy needs.
Recent developments in household collections have been the inclusion of IT use questions in the 2001 Census of Population and Housing and in various ABS social surveys such as Children's Participation in Culture and Leisure Activities. Annual business IT use surveys are now conducted and include topics such as Internet commerce and IT security. Biennial ICT industry surveys are conducted to provide data on ICT production, imports and exports, ICT industry employment, performance and structure information. An Internet activity survey is run every six months and provides regional Internet access details and infrastructure details on Internet service providers.
A major development for 2002–03 will be the compilation of an ICT satellite account. ABS intends to produce an ICT Information Development Plan to guide its future statistical development work in the ICT field.     

Developments in Survey Research over the Past 60 Years: A Personal Perspective

Many developments have occurred in the practice of survey sampling and survey methodology in the past 60 years or so. These developments have been partly driven by the emergence of computers and the continuous growth in computer power over the years and partly by the increasingly sophisticated demands from the users of survey data. The paper reviews these developments with a main emphasis on survey sampling issues for the design and analysis of social surveys. Design‐based inference based on probability samples was the predominant approach in the early years, but over time, that predominance has been eroded by the need to employ model‐dependent methods to deal with missing data and to satisfy analysts' demands for survey estimates that cannot be met with design‐based methods. With the continuous decline in response rates that has occurred in recent years, much current research has focused on the use of non‐probability samples and data collected from administrative records and web surveys.     

Imputation: Methods, Simulation Experiments and Practical Examples

When conducting surveys, two kinds of nonresponse may cause incomplete data files: unit nonresponse (complete nonresponse) and item nonresponse (partial nonresponse). The selectivity of the unit nonresponse is often corrected for. Various imputation techniques can be used for the missing values because of item nonresponse. Several of these imputation techniques are discussed in this report. One is the hot deck imputation. This paper describes two simulation experiments of the hot deck method. In the first study, data are randomly generated, and various percentages of missing values are then non-randomly'added'to the data. The hot deck method is used to reconstruct the data in this Monte Carlo experiment. The performance of the method is evaluated for the means, standard deviations, and correlation coefficients and compared with the available case method. In the second study, the quality of an imputation method is studied by running a simulation experiment. A selection of the data of the Dutch Housing Demand Survey is perturbed by leaving out specific values on a variable. Again hot deck imputations are used to reconstruct the data. The imputations are then compared with the true values. In both experiments the conclusion is that the hot deck method generally performs better than the available case method. This paper also deals with the questions which variables should be imputed and what the duration of the imputation process is. Finally the theory is illustrated by the imputation approaches of the Dutch Housing Demand Survey, the European Community Household Panel Survey (ECHP) and the new Dutch Structure of Earnings Survey (SES). These examples illustrate the levels of missing data that can be experienced in such surveys and the practical problems associated with choosing an appropriate imputation strategy for key items from each survey.     

Longitudinal Surveys and the Study of Occupational Mobility: Panel and Retrospective Design in Comparison

The paper discusses the extent to which longitudinal surveys are able to expose occupational mobility and the dynamics of these processes. It compares two survey designs: the repeated panel design and the retrospective life history design. This comparison details the strength and the weaknesses of the two designs. The paper particularly calls attention to the unique features of the two designs with respect to the exploration of occupational mobility – pointing to three general problems of quantitative social research: item-nonresponse, design effects of single and multi-measurement occasion, and the strong assumption that researchers and respondents share the same definition of concepts. The main conclusion of the paper is that both designs offer unique opportunities to study social change, yet the differences between the two should be kept in mind when choosing the data set for particular research questions.