An Engine-Driven Multi-modal Interaction Framework for Enhancing User Immersion in 3D Virtual Scenes via Smart Gloves and MUFD Algorithms

Abstract

With the rapid development of virtual reality (VR), augmented reality (AR), and other technologies, the interaction experience of three-dimensional virtual scenes has become a hot spot of research. To enhance user immersion, this research proposes an engine-driven multi-modal interaction design for three-dimensional virtual scenes. An interdisciplinary experiment involving 20 volunteers compared multimodal interaction (MMI) and traditional virtual interaction. The interaction process is optimized by combining an intention-capture algorithm, intelligent gloves, a multimodal fuzzy data understanding (MUFD) algorithm, and an axis-aligned bounding box (AABB) collision detection algorithm. The design utilizes an intent capture algorithm for accurately sensing the user's experimental intent, including multiple sources of information such as vision, gesture, and eye tracking. Moreover, a smart glove is used to combine the set of intent probabilities from different channels to more accurately capture the ambiguous or incomplete intent of the user. The results showed that the minimum accuracy for visual interaction was 91.42%. The minimum accuracy for gesture interaction and eye movement interaction was 92.83% and 92.75%, respectively. Compared to traditional virtual interaction, this research method reduced response time by 41.85% and achieved system stability of 99.90%. In terms of immersion, the scores for perceived realism and emotional response were 6.25±1.70 and 5.81±1.67, respectively. Based on the Igroup Presence Questionnaire (IPQ) assessment, the multimodal interaction group showed a 37.96% increase in perceived reality score and a 38.33% increase in emotional response score compared to the traditional group. The study approach received minimal user experience scores of 4.05, 4.26, and 4.84 for naturalness, simplicity of use, and ease of starting, respectively. Furthermore, the system response time took only 120.74ms. In summary, the engine-driven multi-modal interaction design and user immersion enhancement strategy for three-dimensional virtual scenes proposed in this study can significantly enhance users' VR experience. Especially in complex tasks and demanding application scenarios, it can effectively enhance user engagement and satisfaction.