Towards Automated Comprehension Testing of Questionnaire Items using Eye-Tracking

• Applicants: Professorin Dr. Yvonne Kammerer; Dr. Dagmar Kern
• Subject Area: Human Factors, Ergonomics, Human-Machine Systems
• Term: since 2022
• Project identifier: Deutsche Forschungsgemeinschaft (DFG) - Project number 501350073

Questionnaires are important data collection tools used to gather self-report information from individuals. In the social sciences, questionnaires are regularly applied to monitor the society. In psychology, they are commonly used to assess, e.g., individuals’ personality traits, emotional states, or beliefs towards particular issues. Moreover, in relation to temporary events, such as financial crises or the current pandemic, questionnaires are also a suitable method to rapidly assess the population’s opinions and attitudes. To obtain valid and reliable data, however, it is essential to construct questionnaire items (i.e., questions or statements) that are clear and comprehensible for the target group. For this purpose, survey methodologists pretest the items and check whether respondents understand them as intended. A common method to discover problematic items is cognitive interviewing, in which participants verbalize potential comprehension problems. However, the output of such cognitive interviews is bounded, e.g., to the respondents’ ability and willingness to express their thoughts.As an additional unobtrusive method, eye-tracking can be incorporated in pretests to observe respondents’ gaze behavior while completing questionnaires and to detect further problems that otherwise would go unnoticed. However, little is known so far about the relationship between different eye-tracking measures, different kinds of comprehension problems, and the role of individuals’ general reading skills. Furthermore, analyses of eye-tracking data in questionnaire pretests can easily become very complex (depending on the number of considered eye-tracking measures). In this interdisciplinary project, we will contribute to the research field of eye-tracking in questionnaire pretesting by combining expertise in reading research and machine learning. In three experimental user studies, we will collect eye-tracking data of a total of 150 participants (from a quota adult sample) answering a set of 40 questionnaire items. We will analyze the roles of item comprehensibility (experimentally varied) and of general reading skills on participants’ gaze behavior by means of different fine-grained eye-tracking measures. Based on these insights, we will model and train machine learning algorithms to detect comprehension problems automatically. These models will be tested for their generalizability in an evaluation study with another 50 participants. In sum, the results of the proposed project will contribute to a better understanding of how comprehension problems in questionnaire items manifest in respondents’ eye movements, and result in automated methods to make pretests of questionnaire items more efficient and reliable.

• DFG Programme: Research Grants
• Co-Investigators: Professor Dr. Stefan Dietze; Dr. Cornelia Neuert