Computer-Assisted Multimodal Translanguaging Analysis in English Classrooms: A Deep-Learning and NLP Framework

Abstract

Contemporary “all-English” instructional models often marginalize learners’ mother tongues, impeding both comprehension and participation in English-medium classrooms. This paper introduces an end-to-end computer-assisted multimodal framework for the automatic analysis and assessment of translanguaging practices—instances where learners dynamically interweave native-language elements into English discourse. Our system integrates (1) multimodal teaching materials (video, audio, text) designed via expert-driven instructional design; (2) a speech-to-text pipeline for capturing student utterances; (3) NLP preprocessing modules for tokenization, part-of-speech tagging and interlingual code-switch detection; and (4) a fine-tuned BERT-based semantic analyzer that quantitatively scores fluency and naturalness in cross-language segments. We evaluated the platform with a controlled study involving 120 secondary-level learners: the experimental group (multimodal–deep-learning support) outperformed controls by an average of 3.61 % (86 vs. 83 points) across fluency, lexical sophistication and pragmatic appropriateness metrics. These results demonstrate that coupling multimodal interfaces with deep NLP models yields measurable gains in both receptive and productive language skills. Our contributions span cognitive linguistics (quantitative modeling of translanguaging), educational informatics (design of adaptive multimodal content), and applied machine learning (novel BERT-based cross-language evaluator), forging a replicable path for data-driven language teaching and assessment.