Examining the effect of allowing an escape hatch in question scales commonly used in social sciences
Date
2022
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
Many of the scales designed by social scientists to measure the degree to which instincts, information, and intuition guide decision-making lack an opportunity for respondents to express a lack of knowledge or attitude. This study modified six commonly used scales in social-science research by allowing for a “don’t know” (DK) response to determine the effect of providing respondents with an escape hatch. The scales included in this study seek to assess cognitive function, science literacy, or sociocultural values. These scales also vary by response formats (i.e., open-ended, true/false, multiple-choice). To determine the impact of allowing for DK responses, respondents were randomly assigned to either the DK treatment or a control treatment that presented the scales without a DK response option. The survey experiment was created in Qualtrics and distributed to an online panel of 1,520 respondents maintained by Qualtrics. Results indicate heterogeneous effects across the scales and response formats. Overall, the cognitive function and science literacy scales yielded significantly different means for the total number of correct responses when the DK option was available. In addition, true/false questions had the greatest DK response frequency. This study provides insight into the fragility of results of commonly used scales in social-science research.
Description
Keywords
DK, Response format, Scales, Social sciences