The in-depth testing of survey questions: A critical appraisal of methods

It has become increasingly apparent that the conventional piloting, or pre-testing, of survey questions does not ensure that questions are understood by respondents as researchers intend them to be understood. This has led survey methodologists to advocate the use of indepth testing procedures. While some of the procedures used to identify comprehension problems appear to be reasonably effective, almost all of the procedures used to identify other kinds of problems have not proved to be very useful. Two reasons stand out as explanations for this general lack of success. First, the probes used are themselves not likely to be interpreted in the same way by different respondents. Second, the cognitive psychological assumptions that have suggested many of the procedures may not adequately reflect the processes that underlie the way questions are answered. A number of suggestions are made with the view to making the in-depth testing of survey questions a more profitable exercise.