Multimodal Deep Learning Approach for College Students’ Mental Health Monitoring Using Online and Offline Data Integration

Abstract

In response to the difficulty of real-time monitoring and continuous tracking of college students' mental health in the era of new media, we collect the data from online student platforms and offline psychological interviews, and develop a college students' mental health monitoring system based on speech recognition, text extraction, and facial expression recognition, with the goal of achieving intelligent mental health management. The method is to first collect data from students' online network platforms and offline psychological interviews, mainly including multimodal information such as network text data, speech, video images, etc., to study automated speech recognition and text information extraction process methods. At the same time, for the micro expression recognition needs of video images, we propose a VGG19+SE+TA+LSTM network model, which extracts spatial features from four facial regions respectively. VGG19 is used as the convolutional neural network part on the traditional CNN+LSTM network structure, and channel and time attention mechanisms are introduced to enhance the network. The multi region features are fused as the features of a single frame image, and the multi frame image features are input in time series. The long short-term memory network (LSTM) based on time attention mechanism (TA) is used to extract temporal features. Experimental results have shown that integrating multiple modal data from online and offline sources can achieve the automation and intelligence of an intelligent monitoring system for college students' mental health. The fused feature algorithm improves the recognition rates of positive, negative, and neutral emotions by at least 8% and 4.8% respectively compared to the independent Fbank and MFCC feature algorithms, while the VGG19+SE+TA+LSTM network model improves the UF1 evaluation index by nearly 17.9% and 4.5% compared to the CNN+LSTM and VGG19+LSTM models, with providing emotional cognitive references for college students and effectively assisting college counselors in identifying the psychological emotions of college students.