Abstract
Artificial intelligence–driven educational systems have largely prioritised cognitive adaptation, often neglecting the critical role of learners’ emotional states in shaping engagement and learning outcomes. To address this limitation, this study proposes a multimodal, emotion-aware deep learning framework designed to integrate emotional intelligence into intelligent learning environments. The framework jointly analyses facial expressions, speech characteristics, and textual responses to infer learners’ emotional states and models the interdependencies among these modalities through a graph-based fusion mechanism. The proposed approach is evaluated using benchmark emotion datasets, namely AffectNet and IEMOCAP, to assess its capability to recognise emotional patterns and support adaptive feedback during learning interactions. Experimental results demonstrate that incorporating emotional awareness leads to substantial improvements in learner engagement, emotional regulation, and task persistence when compared with conventional cognition-focused systems. The framework achieves consistently high emotion recognition performance, particularly for positive and neutral affective states, and shows robust generalisation across different emotion categories. User study outcomes further suggest that learners perceive the system as more supportive and responsive due to its emotional adaptability. In addition to performance evaluation, the study discusses key ethical considerations associated with emotion-aware educational technologies, including data privacy, informed consent, and responsible deployment. Overall, the findings underscore the potential of multimodal emotional intelligence to advance the development of more empathetic, adaptive, and effective artificial intelligence-based educational systems.
Data availability
The dataset is available from the corresponding author upon individual request.
Abbreviations
- AI:
-
Artificial intelligence
- FER:
-
Facial expression recognition
- NLP:
-
Natural language processing
- CNN:
-
Convolutional neural network
- LSTM:
-
Long short-term memory
- ViT:
-
Vision transformer
- GNN:
-
Graph neural network
- SMOTE:
-
Synthetic minority over-sampling technique
- Dlib:
-
Digital library (face processing toolkit)
- IoT:
-
Internet of Things
- EEG:
-
Electroencephalogram
- GDPR:
-
General data protection regulation
- IEMOCAP:
-
Interactive emotional dyadic motion capture
- SoftMax:
-
Soft maximum function
- EI:
-
Emotional intelligence
- SER:
-
Speech emotion recognition
- DL:
-
Deep learning
- RNN:
-
Recurrent neural network
- TCN:
-
Temporal convolutional network
- BERT:
-
Bidirectional encoder representations from transformers
- MFCC:
-
Mel-frequency cepstral coefficients
- MTCNN:
-
Multi-task cascaded convolutional neural network
- FC:
-
Fully connected
- FL:
-
Federated learning
- HCI:
-
Human–computer interaction
- FERPA:
-
Family educational rights and privacy act
- AffectNet:
-
Facial emotion dataset
- MF:
-
Multimodal fusion
References
Singh, T. M., Reddy, C. K. K., Murthy, B. V. R., Nag, A. & Doss, S. AI and education: Bridging the gap to personalized, efficient, and accessible learning. In Internet of Behavior-Based Computational Intelligence for Smart Education Systems, 131–160 (IGI Global, 2025).
Roumpas, K., Fotopoulos, A. & Xenos, M. A framework for ethical, cognitive-aware human–AI interaction in multimodal adaptive learning systems. In Cognitive-Aware Human–AI Interaction in Multimodal Adaptive Learning Systems.
Islam, M. M., Nooruddin, S., Karray, F. & Muhammad, G. Enhanced multimodal emotion recognition in healthcare analytics: A deep learning-based model-level fusion approach. Biomed. Signal Process. Control 94, 106241 (2024).
Qiang, S. U. N. Deep learning-based modeling methods in personalized education. Artif. Intell. Educ. Stud. 1(1), 23–47 (2025).
Hadinezhad, S., Garg, S. & Lindgren, R. Enhancing inclusivity: Exploring AI applications for diverse learners. In Trust and Inclusion in AI-Mediated Education: Where Human Learning Meets Learning Machines, 163–182 (Springer, Cham, 2024).
Kumar, R., Kumar, P., Sobin, C. C. & Subheesh, N. P. Blockchain and AI in Shaping the Modern Education System (2025).
Lee, A. V. Y., Koh, E. & Looi, C. K. AI in education and learning analytics in Singapore: An overview of key projects and initiatives. Inf. Technol. Educ. Learn. 3(1), Inv-p001 (2023).
Zhou, X. et al. Personalized federated learning with model-contrastive learning for multi-modal user modeling in human-centric metaverse. IEEE J. Sel. Areas Commun. 42(4), 817–831 (2024).
Soman, G., Judy, M. V. & Abou, A. M. Human guided empathetic AI agent for mental health support leveraging reinforcement learning-enhanced retrieval-augmented generation. Cogn. Syst. Res. 90, 101337 (2025).
Xia, B., Innab, N., Kandasamy, V., Ahmadian, A. & Ferrara, M. Intelligent cardiovascular disease diagnosis using deep learning enhanced neural network with ant colony optimization. Sci. Rep. 14(1), 21777 (2024).
Lateef, M. Harnessing AI and machine learning to elevate educational wearable technology. In Wearable Devices and Smart Technology for Educational Teaching Assistance, 53–80. (IGI Global Scientific Publishing, 2025).
Rayudu, K. M., Chinnammal, V., Rubiston, M. M., Padmaloshani, P., Singaravelu, R. & Merlin, N. R. G. Experimental analysis of artificial intelligence powered adaptive learning methodology using enhanced deep learning principle. In 2024 International Conference on Innovative Computing, Intelligent Communication and Smart Electrical Systems (ICSES), 1–7 (IEEE, 2024).
Salloum, S. A., Alomari, K. M., Alfaisal, A. M., Aljanada, R. A. & Basiouni, A. Emotion recognition for enhanced learning: using AI to detect students’ emotions and adjust teaching methods. Smart Learn. Environ. 12(1), 21 (2025).
Vistorte, A. O. R. et al. Integrating artificial intelligence to assess emotions in learning environments: A systematic literature review. Front. Psychol. 15, 1387089 (2024).
Liu, Y. et al. Sample-cohesive pose-aware contrastive facial representation learning. Int. J. Comput. Vis. 133(6), 3727–3745 (2025).
Zhang, X., Cheng, X. & Liu, H. TPRO-NET: an EEG-based emotion recognition method reflecting subtle changes in emotion. Sci. Rep. 14(1), 13491 (2024).
Meng, T. et al. A multi-message passing framework based on heterogeneous graphs in conversational emotion recognition. Neurocomputing 569, 127109 (2024).
Xie, Y., Yang, L., Zhang, M., Chen, S. & Li, J. A review of multimodal interaction in remote education: Technologies, applications, and challenges. Appl. Sci. 15(7), 3937 (2025).
Sangeetha, S. K. B., Immanuel, R. R., Mathivanan, S. K., Cho, J. & Easwaramoorthy, S. V. An empirical analysis of multimodal affective computing approaches for advancing emotional intelligence in artificial intelligence for healthcare. IEEE Access 12, 114416–114434 (2024).
Li, C., Weng, X., Li, Y. & Zhang, T. Multimodal learning engagement assessment system: An innovative approach to optimizing learning engagement. Int. J. Hum. Comput. Interact. 41(5), 3474–3490 (2025).
Khediri, N., Ben Ammar, M. & Kherallah, M. A real-time multimodal intelligent tutoring emotion recognition system (MITERS). Multimed. Tools Appl. 83(19), 57759–57783 (2024).
Sajja, R., Sermet, Y., Cikmaz, M., Cwiertny, D. & Demir, I. Artificial intelligence-enabled intelligent assistant for personalized and adaptive learning in higher education. Information 15(10), 596 (2024).
Chetry, K. K. Transforming education: How AI is revolutionizing the learning experience. Int. J. Res. Publ. Rev. 5(5), 6352–6356 (2024).
Zhang, X. et al. Smart classrooms: How sensors and AI are shaping educational paradigms. Sensors (Basel, Switzerland) 24(17), 5487 (2024).
Govea, J., Navarro, A. M., Sánchez-Viteri, S. & Villegas-Ch, W. Implementation of deep reinforcement learning models for emotion detection and personalization of learning in hybrid educational environments. Front. Artif. Intell. 7, 1458230 (2024).
Yadav, Uma, and Urmila Shrawankar. "Artificial Intelligence Across Industries: A Comprehensive Review With a Focus on Education." AI Applications and Strategies in Teacher Education (2025): 275–320.
Marques-Cobeta, N. Artificial intelligence in education: Unveiling opportunities and challenges. In Innovation and Technologies for the Digital Transformation of Education: European and Latin American Perspectives, 33–42 (2024).
Gan, W., Dao, M. S., Zettsu, K. & Sun, Y. IoT-based multimodal analysis for smart education: Current status, challenges, and opportunities. In Proceedings of the 3rd ACM Workshop on Intelligent Cross-Data Analysis and Retrieval, 32–40 (2022).
Zhou, X., Xuesong, Xu., Liang, W., Zeng, Z. & Yan, Z. Deep-learning-enhanced multitarget detection for end–edge–cloud surveillance in smart IoT. IEEE Internet Things J. 8(16), 12588–12596 (2021).
Halkiopoulos, C. & Gkintoni, E. Leveraging AI in e-learning: Personalized learning and adaptive assessment through cognitive neuropsychology—A systematic analysis. Electronics 13(18), 3762 (2024).
Duan, S., Wang, Z., Wang, S., Chen, M. & Zhang, R. Emotion-aware interaction design in intelligent user interface using multi-modal deep learning. In 2024 5th International Symposium on Computer Engineering and Intelligent Communications (ISCEIC), 110–114 (IEEE, 2024).
Sharma, K., Papamitsiou, Z. & Giannakos, M. Building pipelines for educational data using AI and multimodal analytics: A “grey-box” approach. Br. J. Edu. Technol. 50(6), 3004–3031 (2019).
Villegas-Ch, W., Gutierrez, R. & Mera-Navarrete, A. Multimodal emotional detection system for virtual educational environments: Integration into microsoft teams to improve student engagement. IEEE Access 13, 42910–42933 (2025).
Li, Y., Chai, Z., You, S., Ye, G. & Liu, Q. Student portraits and their applications in personalized learning: Theoretical foundations and practical exploration. Front. Digit. Educ. 2(2), 1–17 (2025).
Javed, S., Ezehra, S. R., Ullah, H. & Naveed, M. How AI can detect emotional cues in students, improving virtual learning environments by providing personalized support and enhancing social-emotional learning. Rev. Appl. Manag. Soc. Sci. 8(2), 665–682 (2025).
Zong, Y. & Yang, L. How AI-enhanced social–emotional learning framework transforms EFL students’ engagement and emotional well-being. Eur. J. Educ. 60(1), e12925 (2025).
Thirunagalingam, A. & Whig, P. Emotional AI integrating human feelings in machine learning. In Humanizing Technology With Emotional Intelligence, 19–32 (IGI Global Scientific Publishing, 2025).
Annapareddy, V. N., Singireddy, J., Nanan, B. P. & Burugulla, J. K. R. Emotional Intelligence in Artificial Agents: Leveraging Deep Multimodal Big Data for Contextual Social Interaction and Adaptive Behavioral Modelling (2025).
Zhang, F., Wang, X. & Zhang, X. Applications of deep learning method of artificial intelligence in education. Educ. Inf. Technol. 30(2), 1563–1587 (2025).
Kolhatin, A. O. From automation to augmentation: a human-centered framework for generative AI in adaptive educational content creation. In CEUR Workshop Proceedings, 143–195 (2025).
Sajja, R., Sermet, Y., Cwiertny, D. & Demir, I. Integrating AI and learning analytics for data-driven pedagogical decisions and personalized interventions in education (2023). https://arxiv.org/abs/2312.09548.
Parkavi, R., Karthikeyan, P. & Abdullah, A. S. Enhancing personalized learning with explainable AI: A chaotic particle swarm optimization-based decision support system. Appl. Soft Comput. 156, 111451 (2024).
Cheng, S., Liu, Q., Chen, E., Huang, Z., Huang, Z., Chen, Y., Ma, H. & Hu, G. DIRT: Deep learning enhanced item response theory for cognitive diagnosis. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, 2397–2400 (2019).
Acknowledgements
The author extends their appreciation to Taif University, Saudi Arabia, for supporting this work through project number (TU-DSPP-2024-17).
Funding
This research was funded by Taif University, Taif, Saudi Arabia, project number (TU-DSPP-2024-17).
Author information
Authors and Affiliations
Contributions
Umesh Kumar Lilhore, Xiaoyu Wu conceptualised the research, designed the methodology, and contributed to data analysis and result interpretation. Tientien Lee was responsible for data collection and played a key role in experimental work while assisting in manuscript drafting and revision. Umesh Kumar Lilhore focused on statistical analysis, data visualisation, and contributed to writing the discussion section. Sarita Simaiya supported laboratory work, experimental processes, and manuscript editing. Roobaea Alroobaea contributed to the literature review and assisted with manuscript revisions. Abdullah M. Baqasah provided technical support during data collection, validated results, and contributed to the methodology. Majed Alsafyani helped with data analysis and interpretation and provided feedback on the manuscript. Finally, Lidia Gosy Tekeste, as the corresponding author, oversaw the project, coordinated the team, and wrote the final manuscript, ensuring the research was completed.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Consent for publication
All authors have reviewed and approved the final manuscript for publication.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Wu, X., Lee, T., Lilhore, U.K. et al. A deep learning approach to emotionally intelligent AI for improved learning outcomes. Sci Rep (2026). https://doi.org/10.1038/s41598-026-37750-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-026-37750-1
