Instructional Manipulation Checks: A longitudinal analysis with implications for MTurk

Instructional Manipulation Checks (IMCs) assess respondent behavior by, for example, asking participants not to answer a trick question. We find IMCs can be used to detect problematic response behavior in longitudinal surveys. This is important because a Latent Class Analysis based on IMC-failure in the two reported studies reveals that between 9% and 12% of respondents can be labelled Inattentive Stayers at the first survey-wave and between 13% and 17% at the third wave. The tendency of Inattentive Stayers to remain in the panel is particularly relevant for online panel services such as MTurk, with workers participating in many surveys over longer time-periods. We find IMC-failure to be mitigated by a warning to respondents that their attention will be checked early in the survey, but not by repeated exposure to IMCs, or a timer leaving questions on the screen longer. Respondent personality also plays a role in IMC-failure, which implies persistency of undesirable respondent behavior.