When often becomes always, and sometimes becomes never: miscomprehension in surveys

Hardy, Ben and Ford, Lucy (2012). When often becomes always, and sometimes becomes never: miscomprehension in surveys. In: Academy of Management Annual Meeting 2012, 3-7 Aug 2012, Boston, MA, US.

DOI: https://doi.org/10.5465/AMBPP.2012.240

URL: http://program.aomonline.org/2012/submission.asp?m...

Abstract

Much of social science involves soliciting the opinions of others, often through the use of surveys. There are a number of different survey formats but a commonly used approach is to provide a stimulus question which is coupled to a limited number of response options. This approach has several benefits. First, the structured format presents the stimulus to each respondent in the same format, reducing inconsistency of stimulus presentation. Second, the standardized format of responses facilitates processing of responses. Third, surveys in this format act as a transduction point where the verbal stimulus of the question is converted to a numerical response through the allocation of ordinal values.

This paper examines miscomprehension and variance in interpretation of survey item through two empirical studies. We find considerable evidence that, whilst respondents understand survey items, they do not all do so in the same way. This variance in comprehension cannot readily be detected statistically, and yet is a significant threat to construct validity, casting doubt on whether the instruments are tapping into the same construct for all respondents.