Nonresponse and Recall Errors in a Study of Absence because of Illness: An Analysis of Their Effects on Distributions and Relationships

Using administrative data as validating standard, we studied the combined effects of two sources of survey error – nonresponse and recall errors – on distributional and substantive bias in a mail survey of absence because of illness among the employees of a Dutch road building company (response rate 77%). No distributional bias was found in five socio-demographic variables (sex, age, years of service, function, and district), but both nonresponse bias and recall bias occurred in our central dependent variables: frequency and duration of absence because of illness. Nonrespondents were on sick leave more frequently and longer than respondents. Furthermore, the self-reports of absence because of illness of our respondents proved to be rather inaccurate. Underreporting of frequency and duration of sick leave was more common than overreporting. Therefore, both sources of error had a cumulative effect.While nonresponse did not result in biased relationships, recall errors had clearly biasing consequences: seven out of 30 correlation coefficients analyzed were too biased to produce valid outcomes; another six were substantially biased. Multiple regression used for predicting recent absence because of illness among our respondents also led to different outcomes depending on the choice of data source (administration or questionnaire) for our absence variables.     

A Middle Class Image of Society

We studied undercoverage and nonresponse in a telephone survey among the population of the City ofGroningen, the Netherlands. The original sample, drawn from the municipal population register,contained 7000 individuals. For 37 percent of them, the telephone company was unable to produce a validtelephone number. Of those with a known telephone number, 49 percent did not answer the telephone orrefused to cooperate. As a result, the final respondents comprised merely 32 percent of the originalsample. To study distributional bias, we used individual-level data compiled from municipal records asour benchmark. Bivariate as well as multivariate analyses showed the undercoverage to be stronglyrelated to all sociodemographic variables studied, except gender. Nonresponse was related to age, countryof origin, marital status, and household size and composition, but not to gender, unemployment, socialassistance benefit, and education. Both undercoverage and nonresponse contributed to a strong middleclass bias in the final data set: middle-aged and older, economically secure people, of Dutch origin andliving with others in a household are overrepresented, while persons in disadvantaged and marginalpositions, such as the young, people of foreign stock, the unemployed, persons depending on publicincome support and singles are underrepresented.     

Survey topic and unit nonresponse

Survey topic as a factor influencing participation rates is becoming increasingly important, as there is a growing trend in social science research for surveying specific populations about specific topics. Previous research has shown that respondents with high topic interest (often referred to as salience) are more likely to participate in surveys. However, the identification of mechanisms that affect respondents’ interest in a survey topic has been largely neglected in research literature. We present an explanatory model of participation that conceptualizes topic interest as a function of an actor’s relational position in a particular social setting. To illustrate the relationship between survey topic and participation behavior, we use an online survey on mating conducted on the user population of an online dating site. For our nonresponse analysis we use web-generated process data, consisting of profile and interaction data, which describe all units of the sample frame. Thus, comprehensive information is available for both participants and non-participants of the online survey on an individual level, enabling a particularly accurate analysis of nonresponse. Results show that the probability of participation varies according to a user’s chances of success on the mating market. Users who can be described as less attractive (e.g. older people, less educated men, overweight women) show a higher probability of participation, which we explain with the mechanism of topic salience. We conclude with general implications regarding (1) the relationship between survey topic and survey participation and (2) the potential of web-generated process data for (online) survey research.     

The power of online panel paradata to predict unit nonresponse and voluntary attrition in a longitudinal design

The objective of this study is to identify factors affecting participation rates, i.e., nonresponse and voluntary attrition rates, and their predictive power in a probability-based online panel. Participation for this panel had already been investigated in the literature according to the socio-demographic and socio-psychological characteristics of respondents and different types of paradata, such as device type or questionnaire navigation, had also been explored. In this study, the predictive power of online panel participation paradata was instead evaluated, which was expected (at least in theory) to offer even more complex insight into respondents’ behavior over time. This kind of paradata would also enable the derivation of longitudinal variables measuring respondents’ panel activity, such as survey outcome rates and consecutive waves with a particular survey outcome prior to a wave (e.g., response, noncontact, refusal), and could also be used in models controlling for unobserved heterogeneity. Using the Life in Australia? participation data for all recruited members for the first 30 waves, multiple linear, binary logistic and panel random-effect logit regression analyses were carried out to assess socio-demographic and online panel paradata predictors of nonresponse and attrition that were available and contributed to the accuracy of prediction and the best statistical modeling. The proposed approach with the derived paradata predictors and random-effect logistic regression proved to be reasonably accurate for predicting nonresponse—with just 15 waves of online panel paradata (even without sociodemographics) and logit random-effect modeling almost four out of five nonrespondents could be correctly identified in the subsequent wave.

     

Attitudes Toward Surveys, Attitude Accessibility and the Effect on Respondents’ Susceptibility to Nonresponse

This paper analyzes whether respondents’ attitudes toward surveys explain their susceptibility to item nonresponse. In contrast to previous studies, the decision to refuse to provide income information, not to answer other questions and the probability of ‘don’t know’ responses is tested separately. Furthermore, the interviewers’ overall judgments of response willingness was included as well. Respondents with a positive and cognitively accessible attitude toward surveys were expected to adopt a cooperative orientation and were thus deemed more likely to answer difficult as well as sensitive questions. Attitudes were measured with a 16-item instrument and the response latencies were used as an indicator for attitude accessibility. We found that respondents with more favorable evaluations of surveys had lower values on all kinds of nonresponse indicators. Except for the strong effect on the prevalence of ‘don’t knows’, survey attitudes were increasingly more predictive for all other aspects of nonresponse when these attitude answers were faster and thus cognitively more accessible. This accessibility, and thus how relevant survey attitudes are for nonresponse, was found to increase with the subjects’ exposure to surveys in the past.     

Is there a 'safe area' where the nonresponse rate has only a modest effect on bias despite non-ignorable nonresponse?

Rising nonresponse rates in social surveys make the issue of nonresponse bias contentious. There are conflicting messages about the importance of high response rates and the hazards of low rates. Some articles (e.g. Groves and Peytcheva, 2008) suggest that the response rate is in general not a good predictor of survey quality. Equally, it is well known that nonresponse may induce bias and increase data collection costs. We go back in the history of the literature of nonresponse and suggest a possible reason to the notion that even a rather small nonresponse rate makes the quality of a survey debatable. We also explore the relationship between nonresponse rate and bias, assuming non-ignorable nonresponse and focusing on estimates of totals or means. We show that there is a ‘safe area’ enclosed by the response rate on the one hand and the correlation between the response propensity and the study variable on the other hand; in this area, (1) the response rate does not greatly affect the nonresponse bias, and (2) the nonresponse bias is small.     

Respondent Related Correlates of Response Behaviour in Audience Research

Audience research by means of surveys has a long tradition, certainly withinarts and humanities oriented research. Yet, due to selective sampling and unitnonresponse it frequently lacks the methodological rigour to make scientificallyvalid statements based on sample estimates. This is one of the first attempts toexplore unit nonresponse in audience research. More specifically, it focuses onthe explanation of nonresponse by the socio-demographic and more topic relatedcharacteristics of a theatre audience. Using a two-step procedure for the on-sitecollection of data, the characteristics of respondents are compared with those ofnonrespondents. In step 1 the composition of the theatre audience is comparedto a proxy of a theatre population benchmark based on a weighted sample of theFlemish population (APS-2000). The validity of this best available method isdiscussed. Step 2 compares respondents with nonrespondents on a micro-level:ignoring unit nonresponse in step 1, we use logistic regression to map selectionin step 2. The chance of cooperating with the survey has been found to increasewith educational attainment and vary according to occupational category. Moreover,involvement with survey topic is confirmed as a strong predictor of survey participation.Gender, age and experience with theatre remain insignificant in predicting responsebehaviour. These findings are compared with the socio-demographic correlates ofresponse behaviour in general populations. Implications for statistically controllingfor nonresponse bias in audience research are discussed. Suggestions for furtherresearch are presented.     

Accuracy of self-reports to the burdensome question: survey response and nonresponse error trade-offs

Survey self-reports to questions involving respondent burden are examined for: (1) overall accuracy, (2) direction of bias, and (3) influence on relationships. Self-reports by respondents to questions on an area of household finances are compared to error-free supplier records. The evidence shows self-reports in this case to be accurate at the overall, or aggregate, level, but inaccurate at the individual, or disaggregate, level. The trade-offs inherent in the nonresponse occurring as a consequence of attempting to acquire verified data versus response bias resulting from a reliance on self-reports from surveys are discussed. It is suggested that attempts by researchers to acquire error-free, verified information in surveys will lead in most cases to unacceptably high levels of item nonresponse. An alternative, algebraic procedure for adjusting for disaggregate level response bias in self-reports on the basis of a subsample of verified responses is provided.     

When neutrality is negative

As the percentage of the population with previous survey experience is high, it is important to study the relation between nonresponse and quality of survey experience. It was hypothesized that the pleasantness of the most recent participation in survey research would influence the willingness of persons to participate again as a respondent in surveys. Results of a survey among 981 respondents in the Netherlands indicate that significantly more respondents with a pleasant most recent survey experience had participated in the past year than respondents with an unpleasant, or indifferent most recent experience. An unpleasant or indifferent most recent participation increased the delay between participations by 50%.     

Asking sensitive questions: the impact of forgiving wording and question context on social desirability bias

Sensitive questions are prone to systematic measurement error due to the respondents?? social desirability concerns. Literature on empirical social research often recommends either positive ??loading?? of sensitive questions, e.g. using ??forgiving?? wording, or manipulating the question context to reduce social desirability bias. We derive theoretical explanations of how manipulations of question wording and context could elicit more socially undesirable answers in sensitive surveys. In an experimental online survey (N?=?1,176), we evaluate the effects of (1) forgiving wording and (2) question context on social desirability bias in different sensitive questions. The empirical evidence on the assumed bias-reducing effects shows inconsistent results. It is indicated however, that the perceived social norm has the strongest and most consistent effect on the respondents?? propensity to self-report socially undesirable behavior.     

The Importance of Experimental Control in Testing the Impact of Interviewer Continuity on Panel Survey Nonresponse

The conventional wisdom in survey research suggests that it is advisable to have the same interviewers return to the same respondents in order to maintain good response rates in longitudinal surveys. There has been, however, very little documented experimental research to support this. Work conducted by Campanelli and O'Muircheartaigh (1999) using a subsample of the British Household Panel Study (BHPS) at Wave 2 with experimental control of the allocation of respondents to interviewers showed no evidence of a positive continuity effect on nonresponse; more extensive analysis by Laurie et al. (1999) of the full BHPS sample using Waves 2 through 4 presents contradictory results. This paper extends the earlier analysis and shows that these differences in findings are due to the lack of experimental control for the inferences from the full BHPS sample in the Laurie et al. (1999) report rather than the shorter time frame considered in Campanelli and O'Muircheartaigh (1999). This paper also considers variation in interviewer continuity effects across areas through the use of multilevel statistical models.     

ALCOHOL MARKETING,ADOLESCENT DRINKING AND PUBLICATION BIAS IN LONGITUDINAL STUDIES: A CRITICAL SURVEY USING META‐ANALYSIS

Abstract This paper presents a meta‐analysis of prospective cohort (longitudinal) studies of alcohol marketing and adolescent drinking, which accounts for publication bias. The paper provides a summary of 12 primary studies of the marketing–drinking relationship. Each primary study surveyed a sample of youth to determine baseline drinking status and marketing exposure, and re‐surveyed the youth to determine subsequent drinking outcomes. Logistic analyses provide estimates of the odds ratio for effects of baseline marketing variables on adolescent drinking at follow‐up. Using meta‐regression analysis, two samples are examined in this paper: 23 effect‐size estimates for drinking onset (initiation); and 40 estimates for other drinking behaviours (frequency, amount, bingeing). Marketing variables include ads in mass media, promotion portrayals, brand recognition and subjective evaluations by survey respondents. Publication bias is assessed using funnel plots that account for ‘missing’ studies, bivariate regressions and multivariate meta‐regressions that account for primary study heterogeneity, heteroskedasticity, data dependencies, publication bias and truncated samples. The empirical results are consistent with publication bias, omitted variable bias in some studies, and lack of a genuine effect, especially for mass media. The paper also discusses ‘dissemination bias’ in the use of research results by primary investigators and health policy interest groups.     

The effect of questionnaire length on survey response

Survey textbooks suggest that long questionnaires should be avoided, and a careful reading of the available empirical evidence confirms the negative effects of substantial length on both response rates and the quality of those responses which are obtained. Data is presented from a lengthy survey in Britain in 1987. Analysis of reasons for nonresponse to this survey suggest that length may indeed have been a significant disincentive to respond for many. However, no effect of length was found on item quality as measured by the number of responses given to open-ended questions. Unexpectedly, the variance in number of responses was greater when the questions were asked later in the questionnaire. The results are interpreted as resulting from the greater power that respondents gain as the survey proceeds.     

Assessing nonresponse bias in activist surveys

How confident can we be that the comparatively low response rates associated with mail surveys of groups of political activists, such as participants of a demonstration, does not hide a substantial nonresponse bias? The paper compares the results of a face-to-face survey of 2003 anti-Iraq war demonstrators in Glasgow, achieving a near perfect response rate, with the data derived from a mail survey handed out to demonstrators eliciting valid responses from 37% of marchers. The comparison shows that better educated, older, female demonstrators were more likely to return the mail questionnaire. Also demonstrators having born a higher ‘cost’ of travelling to the demonstration are more likely to respond. There was no evidence that political interest or political orientation played an important role. However, those who had taken part in demonstrations very frequently in recent years were less likely to return the mail questionnaire. While these results provide some reassurance that even with response rates below 40%, no substantive political bias is present, researchers undertaking surveys of activists should be alerted to the need to address possible nonresponse biases in a systematic way.     

Taking ‘Don’t Knows’ as Valid Responses: A Multiple Complete Random Imputation of Missing Data

Incomplete data is a common problem of survey research. Recent work on multiple imputation techniques has increased analysts’ awareness of the biasing effects of missing data and has also provided a convenient solution. Imputation methods replace non-response with estimates of the unobserved scores. In many instances, however, non-response to a stimulus does not result from measurement problems that inhibit accurate surveying of empirical reality, but from the inapplicability of the survey question. In such cases, existing imputation techniques replace valid non-response with counterfactual estimates of a situation in which the stimulus is applicable to all respondents. This paper suggests an alternative imputation procedure for incomplete data for which no true score exists: multiple complete random imputation, which overcomes the biasing effects of missing data and allows analysts to model respondents’ valid ‘I don’t know’ answers.     

Does survey experience affect respondents’ reported level of satisfaction?

Web surveys are becoming an indispensable tool for quantitative researchers, and online survey panels have proliferated in recent years. However, little research has addressed the challenges of using online panels, namely the potential effects of respondents’ survey experience, also known as panel conditioning. This paper is based on a study of Danish parents’ day care arrangements and their associated level of satisfaction. A survey was conducted through an online panel and included measurements of past survey participation. Through tests of independence on key variables and the application of various ordered logit models, we find no significant evidence that survey experience affects respondents’ reported level of satisfaction. These results persist when testing the potential interaction between survey experience and experiences with day care services. Furthermore, we relate our results to the existing literature and discuss the possibility of different effects cancelling each other out. This leads us to recommendations on the use of online panels and suggestions for elaboration in future research.     

Money donations intentions among Muslim donors: an extended theory of planned behavior model

• The investigation into determinants of money donation intentions while employing an extended theory of planned behavior model is limited to developed country contexts. However, given the challenges facing charitable organizations and scant theoretical evidence from developing world, such an examination can contribute pragmatically. The current study establishes the impact of subjective norms, perceived behavioral control, past behavior, and attitude on respondents' money donation intentions to charities in Pakistan. The respondents (N = 223), a non‐student population living in the city of Gujranwala, completed a survey. The collected data are analyzed by means of a multivariate analysis, which was comprised of regression and correlation. The results reveal a strong support to the extended theory of planned behavior model in establishing the relationship between identified independent and dependent variables in a developing country context of Pakistan. The study contributes to the establishment of a few strategies, which are useful for managers working in charitable organizations to attract and retain donors to support several causes.

Copyright © 2015 John Wiley & Sons, Ltd.     

Bipolarity and/or duality of social survey measurement scales and the question-order effect

When measuring (complex) attitudes within a social survey, researchers often use balanced lists of positive and negative items. The purpose of the present research is to investigate: (a) whether a specific order of measurement scale items can lead to the bipolar (single-dimensional) concept (attitude) being recognised as a dual (bi-dimensional) concept and vice-versa; and (b) whether item order can affect the consistency (metric characteristics) of a measurement scale. An experiment on a group of social science students was conducted: students were randomly split into three subgroups and three different version of a questionnaire (with three differing item orders) were applied. A multi-group confirmatory factor analysis (‘CFA’) and a single group CFA for each item order separately were applied. The final conclusion of the experiment is that there is no general rule about how and when respondents form separate (dual) or unidimensional (continuous) representations of measured concepts. Item-order effects are possible, but they are not as important as one would expect. The results of the experiment also suggest that other factors should be taken into account: the content of the measured concept and the cognitive sophistication of the respondents.     

An econometric method of correcting for unit nonresponse bias in surveys

Past approaches to correcting for unit nonresponse in sample surveys by re-weighting the data assume that the problem is ignorable within arbitrary subgroups of the population. Theory and evidence suggest that this assumption is unlikely to hold, and that household characteristics such as income systematically affect survey compliance. We show that this leaves a bias in the re-weighted data and we propose a method of correcting for this bias. The geographic structure of nonresponse rates allows us to identify a micro compliance function, which is then used to re-weight the unit-record data. An example is given for the US Current Population Surveys, 1998–2004. We find, and correct for, a strong household income effect on response probabilities.     

Tracking precursors of entrepreneurial intention: the case of researchers involved in eco-label industry

This article is based on the adaptation of Ajzen’s theory of planned behavior (TPB) to build an entrepreneurial intention framework tailored to the specific context of researchers involved in eco-label industry, who can be perceived as nascent entrepreneurs. The entrepreneurial intention model is tested on a convenience sample of researchers involved in eco-label industry from a wide range of countries. The configurational effect of research experience, personal attitude, funding instruments and entrepreneurial education level is tested for causal condition related to entrepreneurial intention of 25 eco-label researchers. The qualitative approach of data reveals that research experience and personal attitude relate positively to entrepreneurial career intentions and that these relationships are mediated by entrepreneurial education level. A multi-sided online platform connecting innovators and potential investors is considered a suitable solution for funding respondents’ results of research and innovation activities. Stimulating entrepreneurial motivations and intentions may help researchers to better adapt to alternative career perspectives. Results of this study suggest several ways to stimulate entrepreneurial career choices among researchers involved in eco-label industry.