Respondent Related Correlates of Response Behaviour in Audience Research

Audience research by means of surveys has a long tradition, certainly withinarts and humanities oriented research. Yet, due to selective sampling and unitnonresponse it frequently lacks the methodological rigour to make scientificallyvalid statements based on sample estimates. This is one of the first attempts toexplore unit nonresponse in audience research. More specifically, it focuses onthe explanation of nonresponse by the socio-demographic and more topic relatedcharacteristics of a theatre audience. Using a two-step procedure for the on-sitecollection of data, the characteristics of respondents are compared with those ofnonrespondents. In step 1 the composition of the theatre audience is comparedto a proxy of a theatre population benchmark based on a weighted sample of theFlemish population (APS-2000). The validity of this best available method isdiscussed. Step 2 compares respondents with nonrespondents on a micro-level:ignoring unit nonresponse in step 1, we use logistic regression to map selectionin step 2. The chance of cooperating with the survey has been found to increasewith educational attainment and vary according to occupational category. Moreover,involvement with survey topic is confirmed as a strong predictor of survey participation.Gender, age and experience with theatre remain insignificant in predicting responsebehaviour. These findings are compared with the socio-demographic correlates ofresponse behaviour in general populations. Implications for statistically controllingfor nonresponse bias in audience research are discussed. Suggestions for furtherresearch are presented.     

Survey topic and unit nonresponse

Survey topic as a factor influencing participation rates is becoming increasingly important, as there is a growing trend in social science research for surveying specific populations about specific topics. Previous research has shown that respondents with high topic interest (often referred to as salience) are more likely to participate in surveys. However, the identification of mechanisms that affect respondents’ interest in a survey topic has been largely neglected in research literature. We present an explanatory model of participation that conceptualizes topic interest as a function of an actor’s relational position in a particular social setting. To illustrate the relationship between survey topic and participation behavior, we use an online survey on mating conducted on the user population of an online dating site. For our nonresponse analysis we use web-generated process data, consisting of profile and interaction data, which describe all units of the sample frame. Thus, comprehensive information is available for both participants and non-participants of the online survey on an individual level, enabling a particularly accurate analysis of nonresponse. Results show that the probability of participation varies according to a user’s chances of success on the mating market. Users who can be described as less attractive (e.g. older people, less educated men, overweight women) show a higher probability of participation, which we explain with the mechanism of topic salience. We conclude with general implications regarding (1) the relationship between survey topic and survey participation and (2) the potential of web-generated process data for (online) survey research.     

Does survey experience affect respondents’ reported level of satisfaction?

Web surveys are becoming an indispensable tool for quantitative researchers, and online survey panels have proliferated in recent years. However, little research has addressed the challenges of using online panels, namely the potential effects of respondents’ survey experience, also known as panel conditioning. This paper is based on a study of Danish parents’ day care arrangements and their associated level of satisfaction. A survey was conducted through an online panel and included measurements of past survey participation. Through tests of independence on key variables and the application of various ordered logit models, we find no significant evidence that survey experience affects respondents’ reported level of satisfaction. These results persist when testing the potential interaction between survey experience and experiences with day care services. Furthermore, we relate our results to the existing literature and discuss the possibility of different effects cancelling each other out. This leads us to recommendations on the use of online panels and suggestions for elaboration in future research.     

Nonresponse in panel data: The impact on estimates of a life cycle consumption function

If missing observations in a panel data set are not missing at random, many widely applied estimators may be inconsistent. In this paper we examine empirically several ways to reveal the nature and severity of the selectivity problem due to nonresponse, as well as a number of methods to estimate the resulting models. Using a life cycle consumption function and data from the Expenditure Index Panel from the Netherlands, we discuss simple procedures that can be used to assess whether observations are missing at random, and we consider more complicated estimation procedures that can be used to obtain consistent or efficient estimates in case of selectivity of attrition bias. Finally, some attention is paid to the differences in identification, consistency, and efficiency between inferences from a single wave of the panel, a balanced sub-panel, and an unbalanced panel.     

Unit nonresponse errors in income surveys: a case study

A survey on the economic and social conditions of households in the city of Modena was carried out in 2002 and in 2006 (two waves) by the CAPP (Centre for Analyses of Public Policies). In the first wave of 2002, each designated sampling unit (i.e., the family) had three units as reserves. If the first refused to be interviewed, the interviewer contacted the three reserves, one after the other, until obtaining either one respondent or four non-participant units. At the end of the survey four categories of units were distinguished: interviewees, refusals, noncontacts, and unused reserves. All units were matched with their corresponding record in the databases of the Ministry of Finance of 2002 and the Census of 2001. The resulting data set permitted the analysis of unit or total nonresponses. The distribution of fiscal income showed different shapes for the four categories, implying a selective participation of the families. The interviewees yielded a positive bias of about 600?, holding constant other factors. The selection of the significant factors affecting nonresponse was performed via backward elimination in a logit model and with the lasso method. Participation increased as fiscal income and age increased and by education level (secondary school and university degree), while it decreased among entrepreneurs, independent workers, managers, and medium-to-low skilled workers.     

Panel design effects on response rates and response quality

To understand changes in individuals' opinions and attitudes, it would be best to collect data through panels. Such panels, however, often cause irritation among respondents, resulting in low response rates and low response quality. We address whether this problem can be alleviated by designing a panel survey in an alternative way. For this purpose, we perform two field studies where we measure the effects of several panel design characteristics on response rates and response quality. These characteristics include the number of waves and the time between subsequent waves, which may be either fixed or random. Our findings suggest that response rates and response quality can be improved significantly by surveying at random time intervals. It is then crucial that panel members are not informed about the dates they will be surveyed, because in this case, respondents are less likely to develop expectations as to when they will be surveyed again. The methodology we put forward can be used to improve the efficiency of a panel study by carefully calibrating the studies' panel design parameters.     

Nonresponse and Recall Errors in a Study of Absence because of Illness: An Analysis of Their Effects on Distributions and Relationships

Using administrative data as validating standard, we studied the combined effects of two sources of survey error – nonresponse and recall errors – on distributional and substantive bias in a mail survey of absence because of illness among the employees of a Dutch road building company (response rate 77%). No distributional bias was found in five socio-demographic variables (sex, age, years of service, function, and district), but both nonresponse bias and recall bias occurred in our central dependent variables: frequency and duration of absence because of illness. Nonrespondents were on sick leave more frequently and longer than respondents. Furthermore, the self-reports of absence because of illness of our respondents proved to be rather inaccurate. Underreporting of frequency and duration of sick leave was more common than overreporting. Therefore, both sources of error had a cumulative effect.While nonresponse did not result in biased relationships, recall errors had clearly biasing consequences: seven out of 30 correlation coefficients analyzed were too biased to produce valid outcomes; another six were substantially biased. Multiple regression used for predicting recent absence because of illness among our respondents also led to different outcomes depending on the choice of data source (administration or questionnaire) for our absence variables.     

When neutrality is negative

As the percentage of the population with previous survey experience is high, it is important to study the relation between nonresponse and quality of survey experience. It was hypothesized that the pleasantness of the most recent participation in survey research would influence the willingness of persons to participate again as a respondent in surveys. Results of a survey among 981 respondents in the Netherlands indicate that significantly more respondents with a pleasant most recent survey experience had participated in the past year than respondents with an unpleasant, or indifferent most recent experience. An unpleasant or indifferent most recent participation increased the delay between participations by 50%.     

Nonresponse Bias and Stimulus Effects in the Dutch National Election Study

Bias is a much-debated issue in survey research. Answer effects (respondents claim to have behaved differently than they did in reality), nonresponse bias (nonrespondents differ on important variables from the respondents) and stimulus effects (by participating in a previous wave of a study, respondents change their behavior or attitude) can seriously distort the results of survey research. By using data from the 1998 Dutch National Election Study the authors show that the results of election research can indeed be affected by bias. Not only are significant effects found in the distribution of political attitude and voting behavior variables as a result of both nonresponse bias and stimulus effects, it is also shown that relations between variables change as a result of bias.     

Durations in Panel Data Subject to Attrition: A Note on Estimation in the Case of a Stock Sample1

In the empirical analysis of unemployment durations or job durations, it is generally assumed that the stochastic processes underlying labour market behaviour and the behaviour concerning participation in a panel survey are independent. As a consequence, spells that are incomplete due to attrition can be treated as spells that are subjected to independent right censoring. However, if the assumption of independence is violated, i.e. if for example the probability of dropping out of the panel is related to the rate at which a job is found, then attrition may have to be modelled and estimated jointly with the unemployment duration distribution to avoid biased estimates of the rate at which individuals become employed. A way to model the joint dependence is by means of stochastically related unobserved determinants. We discuss some properties of these kinds of models and state conditions needed to estimate such models in the case of stock sampled duration data.     

New Men and New Women? A Comparison of Paid Work Propensities from a Panel Data Perspective

British Household Panel Survey data for waves 1–5 (1991–5) is used to compare paid work participation rates of men and women. Year-on-year persistence in paid work propensities is high, but greater for men than women. Non-work persistence is higher for women. Using panel data probit regression models, we also investigate why men's and women's participation rates differ, comparing the roles of differences in observable characteristics and differences in rates of return to these characteristics, while also controlling for unobserved heterogeneity. Most of the difference in participation rates is accounted for by the differences in returns associated with the presence of children, especially young ones.     

The effects of interviewer,respondent and area characteristics on cooperation in panel surveys: a multilevel approach

Refusals and noncontacts generally make up the two most important components of unit nonresponse. It is important to separate noncontacts from refusals when examining survey participation, a simultaneously analysis of both components seems relevant when assessing interviewer effects. Using data from a survey of the German Youth Institute Munich, this paper presents an application of the hierarchical regression model, which offers a comprehensive way of analyzing the simultaneous effects of specific interviewer and respondent characteristics. The results provide evidence that the participation on panel surveys is subject to interviewer, respondent as well as area characteristics.     

Multinomial modeling methods: Predicting four decades of international banking crises

This paper examines banking crises in a large sample of countries over a forty-year period. A multinomial modeling approach is applied to panel data in order to track and capture end-to-end cyclical crisis formations, which enhances the binary focus of previous research studies. Several macroeconomic and banking sector variables are shown to be emblematic of leading indicators across the idiosyncratic stages of a banking crisis. Gross domestic product is an early warning signal across all phases, and a concomitant deterioration in consumption spending and fixed capital formation, preceded by a credit boom, signal a banking crisis to come. Currency depreciation exemplifies ensuing financial distress, reinforced by developmental constructs and regional integration. Lower real interest rates, increasing imports, and rising deposits are frequently harbingers of a recovery. Period effects underscore the dynamic evolution of common contemporaneous precursors over time. Premised on pursuing cyclical movements through multiple outcomes, our findings on forecasting performance suggest enhanced predictive power. Several multinomial logistic models generate higher predictive accuracy in contrast to probit models. Compared to machine learning methods (which encompass artificial neural networks, gradient boost, k-nearest neighbors, and random forests methods), a multinomial logistic approach outperforms during pre-crisis periods and when crisis severity is modeled, whereas gradient boost has the highest predictive accuracy across numerous versions of the multinomial model. As investors and policy makers continue to confront banking crises, leading to high economic and social costs, enhanced multinomial modeling methods make a valuable contribution to improved forecasting performance.     

Is there a 'safe area' where the nonresponse rate has only a modest effect on bias despite non-ignorable nonresponse?

Rising nonresponse rates in social surveys make the issue of nonresponse bias contentious. There are conflicting messages about the importance of high response rates and the hazards of low rates. Some articles (e.g. Groves and Peytcheva, 2008) suggest that the response rate is in general not a good predictor of survey quality. Equally, it is well known that nonresponse may induce bias and increase data collection costs. We go back in the history of the literature of nonresponse and suggest a possible reason to the notion that even a rather small nonresponse rate makes the quality of a survey debatable. We also explore the relationship between nonresponse rate and bias, assuming non-ignorable nonresponse and focusing on estimates of totals or means. We show that there is a ‘safe area’ enclosed by the response rate on the one hand and the correlation between the response propensity and the study variable on the other hand; in this area, (1) the response rate does not greatly affect the nonresponse bias, and (2) the nonresponse bias is small.     

Towards a flexible online mixed method design with a feedback loop

The notion of mixed methods design relates to the research studies that combine qualitative and quantitative approaches. However, most of these studies are tailored to specific research problem in a particular study and are typically limited to a fixed sequence of qualitative and quantitative approaches (e.g. qualitative interviews followed by a survey or vice-versa). This limitation historically arises from time, cost and logistic restrictions. As an alternative, we develop an general extension of fixed mixed method design by introducing a flexible feedback-loop so that several phases can be combined in a flexible order. In practice, such designs are now increasingly feasible within an information-communication technology environment, where online respondents are readily available for immediate participation. An online experiment combining interactive series of web surveys and in-depth web interviews was performed to compare this approach with standard two-phase designs involving mixed methods. The costs, timing, quality of finiding and experiences of researchers were systematically evaluated. In summary, the proposed approach proved to be very beneficial.     

On selectivity of nonresponse in discrete-time multi-wave panel studies

The current paper deals with the issue of the detection of selective nonresponse in discrete-time, multi-wave panel studies. If groups in a population differ with respect to the chances that they will be (and remain) in a longitudinal sample, we speak of selective nonresponse. Ultimately, selective nonresponse may lead to a sample that is very different from the target population. We discuss ways to detect and quantify the amount of selectiveness by means of discrete-time Markov models. Then we proceed by addressing how a researcher may gain understanding of how to solve the problems caused by selective nonresponse, and the degree to which these solutions will be effective, by means of data on the nonresponse during a three-wave panel study involving 2800 young Dutch adults.     

Attitudes Toward Surveys, Attitude Accessibility and the Effect on Respondents’ Susceptibility to Nonresponse

This paper analyzes whether respondents’ attitudes toward surveys explain their susceptibility to item nonresponse. In contrast to previous studies, the decision to refuse to provide income information, not to answer other questions and the probability of ‘don’t know’ responses is tested separately. Furthermore, the interviewers’ overall judgments of response willingness was included as well. Respondents with a positive and cognitively accessible attitude toward surveys were expected to adopt a cooperative orientation and were thus deemed more likely to answer difficult as well as sensitive questions. Attitudes were measured with a 16-item instrument and the response latencies were used as an indicator for attitude accessibility. We found that respondents with more favorable evaluations of surveys had lower values on all kinds of nonresponse indicators. Except for the strong effect on the prevalence of ‘don’t knows’, survey attitudes were increasingly more predictive for all other aspects of nonresponse when these attitude answers were faster and thus cognitively more accessible. This accessibility, and thus how relevant survey attitudes are for nonresponse, was found to increase with the subjects’ exposure to surveys in the past.     

Issues in the Design and Analysis of Panel Studies: A Cursory Review

This paper offers a broad review of some aspects in the design and analysis of panel studies, chiefly of household panel surveys. Both the analytic benefits and the potential problems of panel surveys are briefly outlined, and selected methodological and operational issues, which crucially affect data quality are highlighted. These questions are then considered under four headings: (i) dynamic population and its implications for initial sampling and following rules; (ii) panel length and number of waves; (iii) tracking and tracing techniques, and other strategies for maintaining high participation rates; (iv) questionnaire design and strategies for collecting retrospective information. While no technical details are offered, there is some discussion of the possible drawbacks and advantages of the different approaches described.     

A bounds analysis of school completion rates in Australia

Official estimates of school completion rates in Australia increased in the 1980s, peaked in 1992, and fell immediately thereafter before stabilizing. The official estimates were a specific focus of Australian education policy. The decline caused concern at the time. We use data from the Australian Youth Survey (AYS) to gain insight into the behavior of the official estimates. The AYS suffers from nonrepresentativeness, attrition and nonresponse, which means that parameters of interest are not identified. Our bounds analysis is suggestive that school completion was overstated in the official estimates at their peak. Our analysis points to repetition as a key factor in inflating the official estimates. Copyright © 2008 John Wiley & Sons, Ltd.     

Optimal estimates and optimal predictors of finite population characteristics in the presence of nonresponse

Summary When samples from a finite population are studied, for instance by interview, there will usually be some units from which no response is obtained. In this paper optimal predictors of finite population characteristics, when nonresponse is present, are studied. The predictors are studied under simple regression superpopulation models. The optimal predictors are connected to the classical weighted sample estimates which are shown to be maximum likelihood estimates, provided the probability function is fully described by the sampling design. The predictors are compared with respects to their efficiencies for some simple models and a possible explanation to the fact that the poststratification estimate which compensate for nonresponse does no better than the simple estimate, is pointed out.