Middle school teachers’ implementation and perceptions of automated writing evaluation

Abstract

Despite research supporting the efficacy of Automated Writing Evaluation (AWE) in improving writing outcomes, inconsistent implementation by teachers raises concerns about the efficacy of these systems in practice. However, little is known about what factors influence teachers’ implementation and perceptions of AWE. This study examined the relationship between teachers’ implementation and perceptions of the MI Write AWE system, seeking to identify actionable factors that could enhance AWE implementation and acceptance in the future. A mixed-methods design was utilized, combining quantitative analysis of usage logs and survey data with qualitative insights from focus groups and interviews with 19 teachers who participated in a randomized controlled trial (RCT) testing the efficacy of MI Write on students’ writing outcomes. Quantitative data were subjected to descriptive and non-parametric statistical analyses, while qualitative data underwent a deductive coding process, offering an integrated view of MI Write's use and educators’ perceptions. Teachers implemented MI Write variably and not to the extent expected of them within the RCT, but they did report generally positive attitudes towards MI Write. Findings indicated that positive perceptions of system usability and usefulness may be insufficient to promote effective implementation. Instead, ecological factors such as curricular alignment and the challenge of incorporating AWE into existing workload, administrative support, and broader social and educational policy appeared as factors influencing implementation. Findings emphasize that teachers’ implementation and perceptions of AWE are dependent on a range of contextual elements beyond mere system functionality, suggesting that successful adoption requires addressing broader ecological considerations.