Study on decision-making across level of expertise and incentives: an eye-tracking evaluation of a coastal flooding decision support system
Date
2022
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
With the rise in global extreme water events like coastal flooding, the use, and development of early warning systems Decision Support Systems (DSS) have become popular all over the world. A lot of research has been focused on the technical aspect of these systems; however, a technical sound DSS is meaningless if it does not provide a clear, unambiguous, and precise message. Such DSS is used not only by meteorologists but also by many people outside the weather discipline. Incentive, a natural candidate in economic research, has been extensively used to elicit change in human behavior and cognitive effort; however, limited study has been carried out in evaluating the effect of incentive on usability which are also inconclusive. Thus, a lab experiment is conducted to evaluate the usability of the Delaware Coastal Flood Monitoring System (CFMS) across experts and non-experts and people with incentives and non-incentives respectively by incorporating both traditional usability metrics and eye-tracking metrics. We find a significant difference between experts and non-experts in terms of task success rate, time spent on task, fixation count, duration of fixation, and saccades count indicating that experts exert less effort while having a higher task success rate. Non-experts have a significantly low task success rate (p<0.01) with scattered heat maps, especially for the underwater task where people had to interpret a graph and determine the percentage of road that is forecasted to be flooded, compared to experts who have a higher task success rate and more concentrated heat maps. This indicates that some elements of the DSS are too technical for non-experts to understand and need to be improved for better usability. No significant difference was found between the incentive and non-incentive groups across all metrics. Hawthorne effect, an alteration of behavior while being observed, could be one of the reasons for this as the experiment was conducted on a one-to-one basis motivating people to act differently. Overall, this study suggests usability differences between experts & non-experts and provides empirical evidence for those differences in the form of heat maps which serves as a basis for the improvement of DSS. These issues need to be addressed to ensure that these types of DSS can be effectively used to communicate across a diverse audience.
Description
Keywords
Global extreme water events