Multi-Task BERT–BiLSTM–CNN–MHA Framework for Legal Sentiment Analysis and Judgment Prediction

Hongqi Liu

Abstract


This paper proposes a comprehensive multi-task learning (MTL) framework that integrates BERT, Bidirectional Long Short-Term Memory (BiLSTM), Convolutional Neural Network (CNN), and Multi-Head Attention (MHA) to jointly perform legal text sentiment analysis (SA) and predict judicial judgments. The architecture utilizes a shared BERT–BiLSTM encoder to generate contextualized representations, which are then extracted by a CNN module to capture fine-grained sentiment features. Meanwhile, an MHA module fuses information and guides the multi-task output for decision-related subtasks. To ensure effective joint optimization, a weighted loss function is designed to balance task-specific objectives and prevent dominance by any single task. Experiments are conducted on the public CAIL2018 dataset using ten-fold cross-validation to guarantee fairness and reproducibility. In this framework, BERT first encodes legal documents into deep semantic embeddings, which BiLSTM then processes to capture sequential dependencies. The CNN subnetwork extracts localized emotional features from the BiLSTM outputs, achieving 98.0% accuracy and an F1 score of 0.96 on sentiment classification (CAIL-big subset). Simultaneously, the MHA module leverages the shared encoder outputs to highlight and weight relevant legal clauses, supporting downstream tasks such as legal article recommendation and sentencing estimation. The recommendation task achieves 94.0% top-1 accuracy, and sentencing regression achieves a mean squared error (MSE) of 0.028. Compared with traditional baselines such as Word2Vec–BiLSTM and single-task CNN models, the proposed BERT–BiLSTM–CNN–MHA framework delivers over 5% improvement in both sentiment analysis and judgment prediction tasks. Its modular design, deep semantic representation, and robust empirical results validate its effectiveness and practical value for deployment in intelligent legal decision-support and judicial assistance systems.


Full Text:

PDF

References


Krasadakis, P., Sakkopoulos, E., & Verykios, V. S. (2024). A survey on challenges and advances in natural language processing with a focus on legal informatics and low-resource languages. Electronics, 13(3), 648.

Maree, M., Al-Qasem, R., & Tantour, B. (2024). Transforming legal text interactions: leveraging natural language processing and large language models for legal support in Palestinian cooperatives. International Journal of Information Technology, 16(1), 551-558.

DeMattee, A. J. (2023). A grammar of institutions for complex legal topics: Resolving statutory multiplicity and scaling up to jurisdiction‐level legal institutions. Policy Studies Journal, 51(3), 529-550.

Allison, N. G. (2023). From semantic weight to legal ontology via classification of concepts in legal texts. The Law Teacher, 57(2), 201-217.

Royyan, A. R., & Setiawan, E. B. (2022). Feature expansion Word2Vec for sentiment analysis of public policy in Twitter. Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), 6(1), 78-84.

Putra, D. T., & Setiawan, E. B. (2023). Sentiment analysis on social media with glove using combination cnn and roberta. Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), 7(3), 457-563.

Mahto, D., & Yadav, S. C. (2024). Emotion prediction for textual data using GloVe based HeBi-CuDNNLSTM model. Multimedia Tools and Applications, 83(7), 18943-18968.

Bello, A., Ng, S. C., & Leung, M. F. (2023). A BERT framework to sentiment analysis of tweets. Sensors, 23(1), 506.

Khan, L., Amjad, A., Ashraf, N., & Chang, H. T. (2022). Multi-class sentiment analysis of urdu text using multilingual BERT. Scientific Reports, 12(1), 5436.

Abimbola, B., de La Cal Marin, E., & Tan, Q. (2024). Enhancing Legal Sentiment Analysis: A Convolutional Neural Network–Long Short-Term Memory Document-Level Model. Machine Learning and Knowledge Extraction, 6(2), 877-897.

Rajapaksha, I., Mudalige, C. R., Karunarathna, D., de Silva, N., Ratnayaka, G., & Perera, A. S. (2022). SigmaLaw PBSA-A Deep Learning Approach For Aspect Based Sentiment Analysis in Legal Opinion Texts. J. Data Intell., 3(1), 101-115.

Khan, L., Amjad, A., Afaq, K. M., & Chang, H. T. (2022). Deep sentiment analysis using CNN-LSTM architecture of English and Roman Urdu text shared in social media. Applied Sciences, 12(5), 2694.

Qorich, M., & El Ouazzani, R. (2023). Text sentiment classification of Amazon reviews using word embeddings and convolutional neural networks. The Journal of Supercomputing, 79(10), 11029-11054.

Pavan Kumar, M. R., & Jayagopal, P. (2023). Context-sensitive lexicon for imbalanced text sentiment classification using bidirectional LSTM. Journal of Intelligent Manufacturing, 34(5), 2123-2132.

Sirisha, U., & Bolem, S. C. (2022). Aspect based sentiment & emotion analysis with ROBERTa, LSTM. International Journal of Advanced Computer Science and Applications, 13(11).

Zulqarnain, M., Ghazali, R., Aamir, M., & Hassim, Y. M. M. (2024). An efficient two-state GRU based on feature attention mechanism for sentiment analysis. Multimedia Tools and Applications, 83(1), 3085-3110.

Poluru, E., & Syed, H. (2023). A Hybrid Deep Learning GRU based Approach for Text Classification using Word Embedding. EAI Endorsed Transactions on Internet of Things, 10(10.4108).

Zhang, H., Dou, Z., Zhu, Y., & Wen, J. R. (2023). Contrastive learning for legal judgment prediction. ACM Transactions on Information Systems, 41(4), 1-25.

Sengupta, S., & Dave, V. (2022). Predicting applicable law sections from judicial case reports using legislative text analysis with machine learning. Journal of Computational Social Science, 5(1), 503-516.

Alghazzawi, D., Bamasag, O., Albeshri, A., Sana, I., Ullah, H., & Asghar, M. Z. (2022). Efficient prediction of court judgments using an LSTM+ CNN neural network model with an optimal feature set. Mathematics, 10(5), 683.

Esan, A. O. (2024). PERFORMANCE EVALUATION OF MACHINE LEARNING ALGORITHMS FOR JUDICIAL PREDICTION SYSTEM. LAUTECH Journal of Engineering and Technology, 18(1), 192-203.

Hao, S., Zhang, P., Liu, S., & Wang, Y. (2023). Sentiment recognition and analysis method of official document text based on BERT–SVM model. Neural Computing and Applications, 35(35), 24621-24632.

Wicaksono, G. W., Azhar, Y., Hidayah, N. P., & Andreawana, A. (2023). Automatic summarization of court decision documents over narcotic cases using bert. JOIV: International Journal on Informatics Visualization, 7(2), 416-422.

Üveges, I., & Ring, O. (2023). HunEmBERT: a fine-tuned BERT-model for classifying sentiment and emotion in political communication. IEEE Access, 11, 60267-60278.

Jlifi, B., Abidi, C., & Duvallet, C. (2024). Beyond the use of a novel Ensemble based Random Forest-BERT Model (Ens-RF-BERT) for the Sentiment Analysis of the hashtag COVID19 tweets. Social Network Analysis and Mining, 14(1), 88.

Gou, Z., & Li, Y. (2023). Integrating BERT embeddings and BiLSTM for emotion analysis of dialogue. Computational Intelligence and Neuroscience, 2023(1), 6618452.

Yue, W., & Li, L. (2023). Sentiment analysis using a CNN-BiLSTM deep model based on attention classification. International Information Institute (Tokyo). Information, 26(3), 117-162.

Luo, Z. (2024). A study into text sentiment analysis model based on deep learning. International Journal of Information and Communication Technology, 24(8), 64-75.

Wankhade, M., Annavarapu, C. S. R., & Abraham, A. (2024). CBMAFM: CNN-BiLSTM multi-attention fusion mechanism for sentiment classification. Multimedia Tools and Applications, 83(17), 51755-51786.

Başarslan, M. S., & Kayaalp, F. (2023). MBi-GRUMCONV: A novel Multi Bi-GRU and Multi CNN-Based deep learning model for social media sentiment analysis. Journal of Cloud Computing, 12(1), 5.

Erkantarci, B., & Bakal, G. (2024). An empirical study of sentiment analysis utilizing machine learning and deep learning algorithms. Journal of Computational Social Science, 7(1), 241-257.

Jun, W., Tianliang, Z., Jiahui, Z., Tianyi, L., & Chunzhi, W. (2023). Hierarchical multiples self-attention mechanism for multi-modal analysis. Multimedia Systems, 29(6), 3599-3608.

Zhao, S., Liu, R., Cheng, B., & Zhao, D. (2022). Classification-labeled continuousization and multi-domain spatio-temporal fusion for fine-grained urban crime prediction. IEEE Transactions on Knowledge and Data Engineering, 35(7), 6725-6738.

Reyad, M., Sarhan, A. M., & Arafa, M. (2023). A modified Adam algorithm for deep neural network optimization. Neural Computing and Applications, 35(23), 17095-17112.

Zhang, Y., Chen, C., Shi, N., Sun, R., & Luo, Z. Q. (2022). Adam can converge without any modification on update rules. Advances in neural information processing systems, 35, 28386-28399..




DOI: https://doi.org/10.31449/inf.v49i5.9541

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.