Edge-Assisted CNN-Attention Model for Real-Time Multimodal Learner State Recognition in IoT-Enhanced English Language Learning Systems
Abstract
major challenge in computer-assisted English learning lies in accurately perceiving learners’ cognitive and affective states, which limits adaptive feedback and personalized scaffolding. This study proposes an edge-assisted multimodal perception framework that integrates convolutional neural networks with an additive attention mechanism for real-time learner state recognition in IoT-enhanced English classrooms.Multimodal signals, including speech features, facial expressions, gaze, and body posture, are collected through IoT-enabled terminals and preprocessed at edge nodes to reduce latency and communication overhead. CNN layers extract local spatiotemporal features, and the attention module dynamically reweights salient behavioral cues for robust fusion. Learner states such as active oral interaction, passive listening, and distraction are classified, and real-time state outputs support personalized intervention and adaptive task assignment.Experiments on classroom recordings from 128 English learners demonstrated promising performance, achieving over 90% accuracy for major state categories and improved task adaptation efficiency. Results indicate that the proposed edge-assisted multimodal architecture enhances perceptive accuracy and responsiveness, offering a viable pathway toward fine-grained learner modeling and intelligent support in future English language learning environments.References
[1] Ahmed, Rahu Mushtaque. "Integration of wireless sensor networks, Internet of Things, artificial intelligence, and deep learning in smart agriculture: a comprehensive survey: integration of wireless sensor networks, Internet of Things." Journal of Innovative Intelligent Computing and Emerging Technologies (JIICET) 1.01 (2024): 8-19.
https://doi.org/10.1201/9781003107521-7
[2] Hu, Xiaoyan. "The role of deep learning in the innovation of smart classroom teaching mode under the background of internet of things and fuzzy control." Heliyon 9.8 (2023) : 18594-18602 .
https://doi.org/10.1016/j.heliyon.2023.e18594
[3] Shang, Wen-lan. "Application of machine learning and internet of things techniques in evaluation of English teaching effect in colleges." Computational Intelligence and Neuroscience 2022.1 (2022): 7643006 -7643014 .
https://doi.org/10.1155/2022/7643006
[4] Song, Juan. "English Teaching Quality Monitoring and Multidimensional Analysis Based on the Internet of Things and Deep Learning Model." Computational Intelligence and Neuroscience 2022.1 (2022): 9667864 - 96678 73 .
https://doi.org/10.1155/2022/9667864
[5] Sarker, Iqbal H., Asif Irshad Khan, Yoosef B. Abushark , and Fawaz Alsolami. "Internet of things (iot) security intelligence: a comprehensive overview, machine learning solutions and research directions." Mobile Networks and Applications 28.1 (2023): 296-312.
https://doi.org/10.1007/s11036-022-01937-3
[6] Hu, Xiaoying. "Analysis and research on the integrated English teaching effectiveness of internet of things based on stochastic forest algorithm." International Journal of Continuing Engineering Education and Life Long Learning 32.1 (2022): 1-18.
https://doi.org/10.1504/ijceell.2022.121222
[7] Liu, Yamei, and RongQin Li. "Deep learning scoring model in the evaluation of oral English teaching." Computational Intelligence and Neuroscience 2022.1 (2022): 6931796 - 6931 804 .
https://doi.org/10.1155/2022/6931796
[8] Ning, Han. "Design of the Physical Education Teaching System by Using Edge Calculation and the Fuzzy Clustering Algorithm." Mobile Information Systems 2022.1 (2022): 7473614 - 74736 23 .
https://doi.org/10.1155/2022/7473614
[9] Xu, Baoliang. "Application of edge computing and data mining processing system in preschool education courses." Mobile Information Systems 2022.1 (2022): 7872897 - 7872 905 .
https://doi.org/10.1155/2022/7872897
[10] Sun, Zhuomin, M. Anbarasan, and DJCI Praveen Kumar. "Design of online intelligent English teaching platform based on artificial intelligence techniques." Computational Intelligence 37.3 (2021): 1166-1180.
https://doi.org/10.1111/coin.12351
[11] Fitria, Tira Nur. "The use technology based on artificial intelligence in English teaching and learning." ELT Echo: The Journal of English Language Teaching in Foreign Language Context 6.2 (2021): 213-223.
https://doi.org/10.24235/eltecho.v6i2.9299
[12] Fang, Chuanxin. "Intelligent online English teaching system based on SVM algorithm and complex network." Journal of Intelligent & Fuzzy Systems 40.2 (2021): 2709-2719.
https://doi.org/10.3233/jifs-189313
[13] Shan, Qi. "Intelligent learning algorithm for English flipped classroom based on recurrent neural network." Wireless Communications and Mobile Computing 2021.1 (2021): 8020461 - 802046 8 .
https://doi.org/10.1155/2021/8020461
[14] Chen, Shan, and Yingmei Xiao. "An intelligent error correction model for English grammar with hybrid attention mechanism and RNN algorithm." Journal of Intelligent Systems 33.1 (2024): 20230170 - 202301 84 .
https://doi.org/10.1515/jisys-2023-0170
[15] Bai, Yu. "An analysis model of college English classroom patterns using LSTM neural networks." Wireless Communications and Mobile Computing 2022.1 (2022): 6477883 - 64778 92 .
https://doi.org/10.1155/2022/6477883
[16] Geng, Yanmei. "Design of English teaching speech recognition system based on LSTM network and feature extraction." Soft Computing 28.23 (2024): 13873-13883.
https://doi.org/10.1007/s00500-023-08550-w
[17] Huang, Yunlong, and Yanqiu Wang. "The application of graph neural network based on edge computing in english teaching mode reform." Wireless Communications and Mobile Computing 2022.1 (2022): 2611923 - 26119 34 .
https://doi.org/10.1155/2022/2611923
[18] Lilan, Chen, and Jianqi Zhong. "Intelligent recommendation system for College English courses based on graph convolutional networks." Heliyon 10.8 (2024) : 29052-29064 .
https://doi.org/10.1016/j.heliyon.2024.e29052
[19] Chen, Jie, Yukun Chen, and Jiaxin Lin. "Application of Internet of Things intelligent image-positioning studio classroom in English teaching." Journal of high-speed networks 27.3 (2021): 279-289.
https://doi.org/10.3233/jhs-210667
[20] Gao, Wei. "Designing an interactive teaching model of English language using Internet of Things." Soft Computing 26.20 (2022): 10903-10913.
https://doi.org/10.1007/s00500-022-07156-y
[21] Zhou, Ying. "Construction and application of college English multiple intelligence teaching model based on internet of things." Mathematical Problems in Engineering 2022.1 (2022): 5014131 - 50141 40 .
https://doi.org/10.1155/2022/5014131
[22] Li, Yongbin. "Construction of Internet of Things English terms model and analysis of language features via deep learning." The Journal of Supercomputing 78.5 (2022): 6296-6317.
https://doi.org/10.1007/s11227-021-04130-7
[23] Chen, Yafang. "The application of deep learning in the innovation of intelligent English teaching mode." Journal of Computational Methods in Science and Engineering 24.2 (2024): 863-877.
https://doi.org/10.3233/jcm-237054
[24] Ma, Li. "An immersive context teaching method for college English based on artificial intelligence and machine learning in virtual reality technology." Mobile Information Systems 2021.1 (2021): 2637439 - 26374 45 .
https://doi.org/10.1155/2021/2637439
[25] Liu, Yang, and Lei Ren. "The influence of artificial intelligence technology on teaching under the threshold of “Internet+": based on the application example of an English education platform." Wireless Communications and Mobile Computing 2022.1 (2022): 5728569-5728577.
https://doi.org/10.1155/2022/5728569
[26] Jabeen, Summaira, Xi Li, Muhammad Shoib Amin, Omar Bourahla, Songyuan Li, and Abdul Jabbar. "A review on methods and applications in multimodal deep learning." ACM Transactions on Multimedia Computing, Communications and Applications 19.2 (2023): 1-41.
https://doi.org/10.1145/3545572
[27] Li, Yuehua, and Lihao Han. "The role of big data and artificial intelligence in the reform and innovation of intelligent English teaching." Journal of Computational Methods in Sciences and Engineering 24.6 (2024): 3341-3353.
https://doi.org/10.1177/14727978241296745
[28] Wang, Xin, and Simon Smith. "Design of network English autonomous learning education system based on human-computer interaction." Frontiers in Psychology 13 (2022): 989884- 989897.
https://doi.org/10.3389/fpsyg.2022.989884
[29] Meylani, Rusen. "Transforming Education with the Internet of Things: A Journey into Smarter Learning Environments." International Journal of Research in Education and Science 10.1 (2024): 161-178.
https://doi.org/10.46328
[30] Yu, Xiaofang. "Application of IoT Intelligent Distance Education Technology in College English Teaching Reform [J]." International Journal of New Developments in Education 6.8 (2024): 12 -22 .
https://doi.org/10.25236/ijnde.2024.060809
[31] Gui L, Chen Z. A Probabilistic Neural Network-Based Dynamic Perception Framework for Rehabilitation Robots Using Multi-Modal Sensor Fusion[J]. Informatica, 2025, 49(27).
https://doi.org/10.31449/inf.v49i27.8684
[32] Yu S, Li K, Zhao G. An Improved Gated Graph Neural Network for Sports Tourism Recommendation: User Embedded Representations and Attention Mechanisms[J]. Informatica, 2025, 49(27).
https://doi.org/10.31449/inf.v49i27.8490
[33] Shanmugam D B, Dhilipan J. Transformer-Based Model for the Prediction of Sedentary Behavior Patterns Using Deep Learning models[J]. Informatica, 2025, 49(27).
https://doi.org/10.31449/inf.v49i27.7855
[34] Hassanpour A, Yang B. Contactless Vital Sign Monitoring: A Review Towards Multi-Modal Multi-Task Approaches[J]. Sensors, 2025, 25(15): 4792.
https://doi.org/10.3390/s25154792
[35] Henna S, Amjath M, Yar A. Time-generative AI-enabled temporal fusion transformer model for efficient air pollution sensor calibration in IIoT edge environments[J]. AIMS Environmental Science, 2025, 12(3): 526-556.
https://doi.org/10.3934/environsci.2025024
[36] Liu C, Bao L, Zhang Z. A Spatial-Temporal Difference Aggregation Network for Gaofen-2 Multitemporal Image in Cropland Change Area[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024.
https://doi.org/10.1109/jstars.2024.3522066
DOI:
https://doi.org/10.31449/inf.v49i32.10569Downloads
Published
How to Cite
Issue
Section
License
I assign to Informatica, An International Journal of Computing and Informatics ("Journal") the copyright in the manuscript identified above and any additional material (figures, tables, illustrations, software or other information intended for publication) submitted as part of or as a supplement to the manuscript ("Paper") in all forms and media throughout the world, in all languages, for the full term of copyright, effective when and if the article is accepted for publication. This transfer includes the right to reproduce and/or to distribute the Paper to other journals or digital libraries in electronic and online forms and systems.
I understand that I retain the rights to use the pre-prints, off-prints, accepted manuscript and published journal Paper for personal use, scholarly purposes and internal institutional use.
In certain cases, I can ask for retaining the publishing rights of the Paper. The Journal can permit or deny the request for publishing rights, to which I fully agree.
I declare that the submitted Paper is original, has been written by the stated authors and has not been published elsewhere nor is currently being considered for publication by any other journal and will not be submitted for such review while under review by this Journal. The Paper contains no material that violates proprietary rights of any other person or entity. I have obtained written permission from copyright owners for any excerpts from copyrighted works that are included and have credited the sources in my article. I have informed the co-author(s) of the terms of this publishing agreement.
Copyright © Slovenian Society Informatika
