Automated writing evaluation performance: teachers' perceptions of comparative modeling approaches
Date
2024
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
University of Delaware
Abstract
This study uses an experimental comparative design to accomplish two primary goals related teachers’ perceptions of automated writing evaluation (AWE) performance. First, it quantitatively and qualitatively examines teachers’ perceptions of the accuracy and trustworthiness of differentially performing AWE models. Second, it synthesizes interview data about these perceptions to construct a novel framework that defines and explains teachers’ perceptions of high-quality automated feedback. To accomplish these goals, three contrasting AWE models were developed with different machine learning approaches to ensure variation in model performance along predefined dimensions of predictive and descriptive accuracy. Once model performance had been validated, 24 middle school English Language Arts teachers evaluated the comparative performance of the three models during semi-structured interviews where teachers were shown the scores and feedback generated by the models. Observable model performance was represented by a set of randomly selected essays that teachers evaluated prior to the interview. This study found that teachers were sensitive to detecting differences in AWE model performance. Teachers were more likely to perceive automated scores and feedback as accurate when the model output was more similar to their own. Further, exposure to and reflection on AWE model performance significantly increased teachers’ trust in AWE. Finally, the content analysis of interview transcripts revealed that there are six key characteristics that explain teachers’ perceptions of high-quality automated feedback.
Description
Keywords
Assessment, Automated writing evaluation, Machine learning, Natural language processing, Perceptions