KorvexChecker: A BERT-Based Verification Framework for LLM Outputs in Turkish

Abstract

Large Language Models (LLMs) have demonstrated exceptional capabilities in generating human-like text, yet their outputs often suffer from critical issues such as hallucination, ethical violations, and lack of meaningfulness. These challenges can result in misinformation, offensive language, and unreliable outputs, especially in niche languages like Turkish. To address these concerns, we introduce KorvexChecker, a modular and multi-functional tool for validating LLM outputs in terms of ethical compliance, meaningfulness, and hallucination detection. KorvexChecker employs a combination of fine-tuned BERT models for sentiment analysis, ethical violation detection, and text classification to identify whether a given text requires further verification. The tool integrates various external verification resources, such as Web Search API for factual accuracy and journal database for academical document validation. Furthermore, Zemberek, a Turkish NLP library, is utilized for preprocessing, normalization, and segmentation of Turkish text. The system was evaluated on a custom-labeled dataset of 7552 Turkish samples, divided into training, validation, and test sets. Experimental results demonstrate that the BERT-based classifiers achieved an average F1-score of 93.3% across multiple tasks, significantly outperforming both rule-based methods and traditional machine learning baselines. A small-scale user test showed the framework to be responsive and effective in real-time scenarios, with an average inference latency of 800 milliseconds per query. By leveraging both advanced machine learning techniques and external validation mechanisms, KorvexChecker offers a robust framework for ensuring the trustworthiness of AI-generated text, with potential applications in academia, media, and content moderation.