METCL-BERT: A BERTScore and Contrastive Learning-Based Framework for Automatic Translation Quality Assessment of Large Language Models

Abstract

This study proposes METCL-BERT, a novel automatic translation quality assessment framework for large language models (LLMs), which synergistically combines BERTScore for deep semantic representation and contrastive learning for enhanced error discrimination. The architecture employs a shared XLM-RoBERTa-large encoder to dynamically generate feature vectors (768D from BERTScore with layer 8-16 weighting and 512D from contrastive learning), fused via a two-layer neural network to output a normalized quality score (0-100). Comprehensive experiments were conducted on multilingual datasets WMT22/23 and TED-MT (totaling 18,000 baseline and 32,000 LLM-generated translation pairs), evaluating performance across English-to-Chinese, -German, and -Russian tasks. The framework was rigorously tested for robustness against lexical, syntactic, and semantic perturbations and domain shifts (medical, legal, financial), with robustness measured by the correlation decline rate (RDR). Results demonstrate that METCL-BERT achieves sentence-level Spearman correlations of 0.791 (en-zh), 0.803 (en-de), and 0.782 (en-ru), significantly outperforming the best baseline KIWI-22 by >7.6%. It attains a system-level Kendall Tau of 0.832, markedly superior to COMET-22 (0.745). Crucially, its robustness is validated by an average RDR of 18.70% across perturbation tests, substantially lower than BERTScore (24.50%) and COMET-22 (21.20%). Further strengths include exceptional discriminative power (QSD=2.14) with strictly increasing quality interval medians (92.5→78.0→65.0→38.0) and a large effect size (Cohen's d=4.37). Ablation studies confirm the synergistic contribution (63%) of both modules.