ESVA: Enhancing Multimodal Emotion Recognition via Multi-Scale Audio Feature Extraction and Cross-Modal Temporal Alignment

Abstract

Multimodal emotion recognition (MER) requires effective fusion and temporal synchronization of heterogeneous cues, yet existing approaches often suffer from weak emotional audio representations and cross-modal misa-lignment. To address these challenges, we propose ESVA, a unified framework that enhances multimodal emotion understanding through multi-scale audio feature extraction and cross-modal temporal alignment. Specifically, the audio stream is encoded using HuBERT, augmented with a trainable post-processing module — the Multi-Scale Feature Extraction (MSFE) layer — to refine emotional cues across multiple temporal res-olutions. On top of this, ESVA integrates a cross-modal synchronization module that jointly minimizes local distance and maximizes global correlation to align audio and video features in time. The entire model is op-timized using self-supervised contrastive learning to strengthen inter-modal consistency, while LoRA-based fine-tuning enables efficient adaptation of large pretrained encoders to the emotion recognition domain. Ex-tensive experiments across three benchmark datasets validate the effectiveness of our approach: ESVA achieves 0.9074 F1 on MER2023, 0.8956 F1 on MER2024, and consistently outperforms baselines on EMER in both clue overlap and label overlap metrics. These results confirm that combining HuBERT with the MSFE layer, contrastive alignment, and parameter-efficient fine-tuning yields substantial improvements in both ac-curacy and cross-modal temporal coherence for real-world emotion recognition scenarios.