Using screen video capture software to aide and inform cognitive interviewing

Web-based surveys are a salient tool in the repertoire of social and behavioral scientists. The increase in web-based surveys is understandable considering the distinct advantages offered, including: (a) lower costs and reduced labor time, (b) ability to directly transfer data into statistical packages (reducing coding errors), (c) customization options enabling more attractive presentation, (d) ability to reduce respondent burden by embedding skip patterns, and (e) access to larger sample sizes in different geographic regions. It is important to note, however, that administering web-based surveys also introduces distinct sources of error (e.g., coverage, sampling and non-response). Regardless of format (e.g., paper-and-pencil or web-based), specific, prescribed steps must be followed when constructing an instrument in order to reduce survey error and lend credence to the data collected before subsequent analysis is performed. One of those crucial stages integral to the pretesting process is cognitive interviewing. Cognitive interviewing is a qualitative process, encompassing two main techniques: think aloud interviewing and verbal probing. Collectively, these two methods seek to (a) produce information on what the respondent is thinking while answering the questions, (b) the cognitive processes used to answer the questions, and (c) how the respondent answers the questions. The purpose of this article is to provide a practical guide outlining how Camtasia, a screen video capture software, can aide and inform the cognitive interview process.     

Does Repeated Measurement Improve Income Data Quality?

This paper exploits a natural experiment created by a survey design to show that the quality of income data systematically changes across waves of a panel. We estimate that the effect of being interviewed for a second time, relative to the first, is to increase mean monthly income by 8%. Dependent interviewing – a recall device commonly used in panel surveys – explains one third of the observed increase. The remaining share is attributed to changes in respondent behaviour (panel conditioning). We review the evidence for and against a reporting improvement vs. a behavioural response by survey participants.     

Question-Answer Sequences in Survey-Interviews

Interaction analysis was used to analyze a total of 14,265 question–answer sequences of (Q-A Sequences) 80 questions that originated from two face-to-face and three telephone surveys. The analysis was directed towards the causes and effects of particular interactional problems. Our results showed that problematic respondent behavior is affected by the questionnaire design, whereas inadequate interviewer behavior is affected by respondent behavior, rather than directly by the questionnaire design. Two surveys used questions for which validating information was available. It appeared that the occurrence of such irregularities of interviewer and respondent behavior was related to the validity of the eventual responses. Explanations for the occurrence of problematic respondent behavior were proposed, concerning both cognitive and conversational factors, related to the wording of questions and response alternatives. Part of this paper was presented at the International Conference on Questionnaire Development, Evaluation and Testing Methods, 14–17 November 2002, Charleston, South Carolina.     

Response Effects in Surveys on Children and Adolescents: The Effect of Number of Response Options,Negative Wording,and Neutral Mid-Point

Social researchers increasingly survey children and young adolescents.They are convinced that information about perspectives, attitudes, and behaviors of childrenshould be collected from the children themselves. Methodological expertise on surveying childrenis still scarce, and researchers rely on ad-hoc knowledge from fields such as child psychiatry and educationaltesting, or on methodological knowledge on surveying adults. Regarding adults, empirical evidenceshows that respondent characteristics (cognitive abilities) as well asquestion characteristics (question difficulty) affect response quality.This study reports on a methodological survey experiment on theeffect of negatively formulated questions, the number of response options and offeringa neutral midpoint as response option question characteristics on the reliability of theresponses, using children and young adolescents as respondents.The study shows no effects of negatively formulated questions onthe reliability measures, although children respond consistently differently on negativelyformulated questions than on positively formulated questions. Taking all results on the effectsof number of response options and offering a neutral midpoints on the different reliabilitymeasures into consideration; it would appear that offering about four responseoptions is optimal with children as respondents.     

Children between the age of 5 and 11: what “don’t know” answers tell us

Children are increasingly seen as competent informants on their activities, views and needs. Different abilities of children can lead to inadequate (formally or regarding the content) answers or to the misunderstanding of proceedings during an interview process. One inadequate or at least unintended (on the part of the researcher) response is a “don’t know” answer (DK). To study when and why DK answers occur, a possible cause for them in the cognitive process and their relation to age, we analysed the DK answers given in qualitative semi-structured interviews with children aged 5–11, conducted face-to-face and on the telephone. Each of 56 children was interviewed twice, once face-to-face and once via telephone. The results show that age had an effect on DK responses due to the cognitive state of the respondent and their communicative intent, but there was no difference in the number of DKs due to problems with the research instrument or due to an adequacy judgement in respect of respondent age.     

Accuracy of self-reports to the burdensome question: survey response and nonresponse error trade-offs

Survey self-reports to questions involving respondent burden are examined for: (1) overall accuracy, (2) direction of bias, and (3) influence on relationships. Self-reports by respondents to questions on an area of household finances are compared to error-free supplier records. The evidence shows self-reports in this case to be accurate at the overall, or aggregate, level, but inaccurate at the individual, or disaggregate, level. The trade-offs inherent in the nonresponse occurring as a consequence of attempting to acquire verified data versus response bias resulting from a reliance on self-reports from surveys are discussed. It is suggested that attempts by researchers to acquire error-free, verified information in surveys will lead in most cases to unacceptably high levels of item nonresponse. An alternative, algebraic procedure for adjusting for disaggregate level response bias in self-reports on the basis of a subsample of verified responses is provided.     

Gossip in the workplace and the implications for HR management: a study of gossip and its relationship to employee cynicism

Gossip is a common phenomenon in the workplace, but yet relatively little is understood about its influence to employees. This study adopts social information theory and social cognitive theory to interpret the diverse literature on gossip, and to develop and test hypotheses concerning some of the antecedents of gossip, with an aim of developing knowledge of the relationship between gossip and employee behaviour in the workplace. The study analysed survey data in a two-stage process, from 362 employees across a range of industries in Taiwan. The findings revealed that job-related gossip predicted employee cynicism and mediated the relationship between psychological contract violation and cynicism, and that non-job-related gossip showed a similar but weaker effect to employee cynicism. The contribution made by this paper is of value to both the academic subject domain and managers in Human Resources. First, we have identified two constructs of gossip, job-related and non-job-related gossip not previously reported and a validated scale has been created. Second, we have confirmed that these different constructs of gossip impact differently on employee behaviour and therefore HR managers should be cautious about gossip in the workplace, as it can cause cynical behaviour amongst employees.     

Learning and fatigue during choice experiments: a comparison of online and mail survey modes

This study investigates the effect of survey mode on respondent learning and fatigue during repeated choice experiments. Stated preference data are obtained from an experiment concerning high‐speed Internet service conducted on samples of mail respondents and online respondents. We identify and estimate aspects of the error components for different subsets of the choice questions, for both mail and online respondents. Results show mail respondents answer questions consistently throughout a series of choice experiments, but the quality of the online respondents' answers declines. Therefore, while the online survey provides lower survey administration costs and reduced time between implementation and data analysis, such benefits come at the cost of less precise responses. Copyright © 2008 John Wiley & Sons, Ltd.     

TRAINING IN PERFORMANCE APPRAISAL INTERVIEWING: AN EVALUATION STUDY

The paper describes a study of the effects of a training course in performance appraisal interviewing. Managers who had attended the course were asked, by means of a postal questionnaire, to compare their pre-training and post-training interviewing performances. There were three important findings. First, that the trainees had improved on almost every aspect of appraisal interviewing; secondly, it is not just the skill of interviewing which is important, but also an understanding of the role of performance appraisal; and finally, managers in mid-career may have most to gain from training of this type.     

Managerial attitudes toward environmental management within Australia,the People's Republic of China and Indonesia

This study presents a survey of the attitudes of corporate managers and managerial students across Australia, the People's Republic of China and Indonesia toward 18 key contemporary environmental management issues. The study sought to explore whether respondents from these countries, characterized by differing levels of development, also differ in their attitude toward environmental management. Results indicated that, despite age being a moderating factor, significant differences did exist between the 676 country respondents on 15 of the 18 questions. Contrary to expectations, Australian respondents were more cautious of supporting a forthright view on environmental issues, whilst Chinese respondents favoured a more centralized approach to decision making regarding the environment. The results lend marginal support to the new environmental paradigm (NEP), but also to the radicalization of environmental issues and age as a possible influence on respondent beliefs. Copyright © 2006 John Wiley & Sons, Ltd and ERP Environment.     

Print versus electronic surveys: a comparison of two data collection methodologies

This paper compares the responses of consumers who submitted answers to a survey instrument focusing on Internet purchasing patterns both electronically and using traditional paper response methods. We present the results of a controlled experiment within a larger data collection effort. The same survey instrument was completed by 416 Internet customers of a major office supplies company, with approximately 60% receiving the survey in paper form and 40% receiving the electronic version. In order to evaluate the efficacy of electronic surveys relative to traditional, printed surveys we conduct two levels of analysis. On a macro-level, we compare the two groups for similarity in terms of fairly aggregate, coarse data characteristics such as response rates, proportion of missing data, scale means and inter-item reliability. On a more fine-grained, micro-level, we compare the two groups for aspects of data integrity such as the presence of data runs and measurement errors. This deeper, finer-grained analysis allows an examination of the potential benefits and flaws of electronic data collection.Our findings suggest that electronic surveys are generally comparable to print surveys in most respects, but that there are a few key advantages and challenges that researchers should evaluate. Notably, our sample indicates that electronic surveys have fewer missing responses and can be coded/presented in a more flexible manner (namely, contingent coding with different respondents receiving different questions depending on the response to earlier questions) that offers researchers new capabilities.     

Why don’t they answer? Unit non-response in the IAB establishment panel

The paper proposed focuses on the unit-nonresponse in the IAB (Institute for Employment Research) Establishment Panel, a comprehensive data set describing the employer side of the labour market in Germany. Every year since 1993 (1996) the IAB Establishment Panel has surveyed the same establishments from all branches and different size categories in western (eastern) Germany. Although great efforts are taken to convince the owner/manager to take part in the survey there are always firms that do not answer the questionnaire. In this paper the authors try to find out why some establishments are not willing or able to respond to the questionnaire. If the respondent has the authority to provide relevant information, is able to give reliable answers to the questions with a justifiable amount of effort and is interested in the survey in business terms, participation is less frequently refused. The results also confirm the central significance of the interaction between the respondent and the interviewer. If one of the two individuals changes, the probability of further participation falls clearly.     

Cultural Readjustment After Crisis: Regulation and Learning from Crisis Within the UK Soccer Industry*

abstract Challenging Turner's (1976 , 1978 ) implicit assumption that cultural readjustment typically follows a crisis, this paper examines the evolution of the regulation of safety management within the UK soccer industry since 1946. Employing a longitudinal case study approach, the industry's response to four crises is examined. This study explores the industry's response to changes in the regulatory framework, through the lens of Gouldner's (1954 ) patterns of industrial bureaucracy and institutional theory. The persistence of indulgent and mock patterns of behaviour, following a series of disasters, challenges Turner's implicit assumption. Although a move towards a punitive approach to regulatory behaviour effected some changes, there was limited evidence of cultural readjustment due in part to a ‘mindset of invulnerability’ ( Wicks, 2001 ). This included the fixed belief that hooligan behaviour was the primary problem facing the industry. Our findings suggest that more participative forms of regulation encourage more effective learning from crisis because they challenge core organizational and individual assumptions.     

Results of a cross-national structured cognitive interviewing protocol to test measures of disability

This paper describes the Washington Group project to test a short battery of disability questions developed for national censuses. The study used an unusually structured cognitive test protocol and was administered to a total of 1,290 respondents selected from convenience samples in fifteen countries in Central and South America, Asia and Africa. The test protocol consisted of the six core disability questions followed by questions designed to illustrate: (1) whether core questions were administered with relative ease; (2) how core questions were interpreted by respondents; (3) the factors considered by respondents when forming answers to core questions; and (4) the degree of consistency between responses to core questions and a set of more detailed functioning questions. Additionally, demographic and general health sections allowed for an examination of comparability, specifically, whether test questions performed consistently across all respondents, or if nationality, education, gender or socio-economic status impacted the ways in which respondents interpreted or considered each core question.     

Social Distance,Respondent Cooperation and Item Nonresponse in Sex Survey

This paper examines the ways in which the number of item nonresponses is determined by social distance and/or interview rapport, with a focus on responses of “refusal” and “don’t know”, implying the respondent’s lack of willingness and ability to provide substantive responses to sensitive questions. The data analyzed were from 39 self- administered questions concerning sexual attitudes and behaviors in the 2002 Taiwan Social Change Survey for module “Family and Changing Gender Role”. Poisson Regression in 2-level Hierarchical Linear Model was employed to enhance the accuracy of the analysis of the accumulation of “don’t know” and “refusal” responses. The results showed that respondent cooperation significantly decreased the number of both “don’t know” and “refusal” replies. The decrease was not conditioned by any kind of social distance. Age and education distances have respectively negative and positive effect on the number of “don’t know” and “refusal” answers. The married–married interview produced more “don’t know” and “refusal” than other paired interview types. The larger the ethnicity distance is, the more “refusal” appears. The substantial findings imply that the effects of social-distance and rapport (respondent cooperation) on the number of item nonresponses deserve more attention in research on survey methodology. The divergent findings on gender-distance effect and marital-status effect, however, call for replication studies in the future.     

On possibilities and limitations of using self-anchoring scales in web surveys

Self-anchoring scales were first mentioned by Kilpatrick and Cantril (J Indiv Psychol 16:158–170, 1960) and Cantril (The pattern of human concerns, 1965) as rating instruments in which the end anchors are defined by the respondent himself, basing on his own assumptions, perceptions, goals and values. The uses of these scales are legion and they have shown to be very useful in reducing measurement bias in cross-cultural research (Cantril, The pattern of human concerns, 1965; Bernheim et al. J. Happiness Stud. 7:227–250, 2006). The first part of the current study investigates whether context effects can be lessened or eliminated by using self-anchoring scales. For this purpose, an experiment similar to the ones by Couper et al. (Public Opin Q 71:623–634, 2004, Public Opin Q 68:255–266, 2007), in which they manipulated images that figured in a web survey, was conducted. The hypothesis that self-anchoring scales can reduce contextual bias, is not supported by our data. The second part of the study investigates if and how self-anchoring scales affect drop-out during the filling-out of questionnaires. It is found that, compared to a regular rating scale, a larger proportion of respondents drop-out. Moreover, subjective preferences for the one or the other scale do not seem to differ.     

Preventing mismatch answers in standardized survey interviews

Interaction analysis of question–answer sequences from a telephone survey shows that so-called mismatch answers, i.e. answers that do not correspond to the required answering format, are the most frequently occurring problematic verbal behavior. They also are likely to trigger suggestive interviewer probing. Explanations for the occurrence of mismatch answers concern cognitive and conversational factors. In both a non-experimental and an experimental study it was found that questions with formal response alternatives yield more mismatch answers than questions with colloquial response alternatives (i.e., words that are frequently used in ordinary conversations). Effects of the wording of questions were only found in the non-experimental study, indicating that formal question wording yields fewer mismatch answers than colloquial question wording. The findings suggest that, especially in case of questions that are formulated as agree–disagree opinion assertions, the chance of mismatch answers is highly reduced when colloquial response alternatives are used.     

The Structure and Management of Alliances: Syndication in the Venture Capital Industry*

abstract Syndicates are a form of inter‐firm alliance in which two or more venture capital firms co‐invest in an investee firm and share a joint pay‐off. Syndication is a significant part of the venture capital market yet little research has been conducted into the process of structuring syndicate deals and the management of syndicates following deal completion. This paper analyses the neglected issues concerning the structuring and management of syndicated venture capital investments from the perspectives of both lead and non‐lead syndicate members using two surveys of venture capital firms and examination of syndication documents. Lead investors typically have larger equity stakes and the syndicated investment agreement is a document that enshrines the rights of participants rather than specifying behaviour. Contractual arrangements typically serve as a back drop to relationships as non‐legal sanctions are important and decisions are typically reached following discussion and consensus, but lead venture capital investors’ residual and specific powers are important in ensuring timely decision‐making. The findings extend previous work on alliances by emphasizing the importance of non‐legal sanctions, especially reputation effects, in mitigating opportunistic behaviour by dominant equity holders. The paper also adds to the limited research on the dynamics of alliances by highlighting the role of repeat syndicates.     

新乡市民营企业家成功要素研究

通过对政府备案资料和公开资料的统计,对一些民营企业家实地采访,以及通过放开式问卷调查法,获得确实数据,发现涉及的民营企业家成功要素,然后进行合并相近,相似的指标,最后再进行频数统计。总结出前几项所占频数较多的成功要素,从中得出新乡民营企业家成功的独特要素。