Hybrid Deep Learning Model to Estimate Cognitive Effort from fNIRS Signals
Date
2025-10-12
Journal Title
Journal ISSN
Volume Title
Publisher
Companion Proceedings of the 27th International Conference on Multimodal Interaction
Abstract
This study estimates cognitive effort based on functional near-infrared spectroscopy data and performance scores using a hybrid DeepNet model. The estimation of cognitive effort enables educators to modify material to enhance learning effectiveness and student engagement. In this study, we collected oxygenated hemoglobin using functional near-infrared spectroscopy during an educational quiz game. Participants (n=16) responded to 16 questions in a Unity-based educational game, each within a 30-second response time limit. We used DeepNet models to predict the performance score from the oxygenated hemoglobin, and compared traditional machine learning and DeepNet models to determine which approach provides better accuracy in predicting performance scores. The result shows that the proposed CNN-GRU gives better performance with 73% than other models. After the prediction, we used the predicted score and the oxygenated hemoglobin to observe cognitive effort by calculating relative neural efficiency and involvement in our test cases. Our result shows that even with moderate accuracy, the predicted cognitive effort closely follow the actual trends. This findings can be helpful in designing and improving learning environments and provide valuable insights into learning materials.
Description
This article was originally published in ICMI '25 Companion: Companion Proceedings of the 27th International Conference on Multimodal Interaction. The version of record is available at: https://doi.org/10.1145/3747327.3764901
This work is licensed under a Creative Commons Attribution 4.0 International License. https://creativecommons.org/licenses/by/4.0/ ICMI Companion ’25, Canberra, ACT, Australia © 2025 Copyright held by the owner/author(s).
Keywords
Deep learning, cognitive effort, relative neural efficiency, relative neural involvement, performance score, functional Near-Infrared Spectroscopy (fNIRS), brain signal, hemodynamic response, educational games
Citation
Sharmin, S., & Barmaki, R. L. (2025). Hybrid Deep Learning Model to Estimate Cognitive Effort from fNIRS Signals. Companion Proceedings of the 27th International Conference on Multimodal Interaction, 227–234. https://doi.org/10.1145/3747327.3764901