Transformer-Based Fake News Classification: Evaluation of DistilBERT With CNN-LSTM and GloVe Embedding
Abstract
Social media networks have changed the face of communication in recent times, but at the same time, they have brought various challenges, such as disseminating fake news and information. NLP and machinelearning algorithms try to meet these challenges by structuring online information, although dataset bias remains a critical concern. SA has helped people gain insight into the context of news dissemination. Still, sham news dissemination-often by fake accounts represents a great hazard not only to users but to the stability of society. Several researchers have tried to assess the credibility of information and reduce sham data flow. In this work, the datasets of 17,903 fake news and 20,826 real news from Kaggle will be used. Preprocessing steps included text normalization, removal of punctuation, links, usernames, and nonalphabetic characters to prepare the data for analysis. This study explored categorizing fake and true news using advanced NLP techniques, transformer-based architectures, and deep learning models. A focus on improving classification accuracy and addressing dataset bias was achieved through models like DistilBERT, CNN, and LSTM. DistilBERT demonstrated remarkable performance, achieving an accuracy of 99.65%, with precision, recall, F1-score, and ROC-AUC values of 0.992188, 1, 0.996078, and 0.996894, respectively, outperforming the other models. The study's novelty lies in its detailed evaluation of DistilBERT, which showed significant improvements in accuracy, recall, and AUC while mitigating dataset bias. The results highlight the potential of DistilBERT for robust and reliable fake news classification, addressing critical limitations in existing approaches.References
A. Wani, I. Joshi, S. Khandve, V. Wagh, and R. Joshi, “Evaluating deep learning approaches for covid19 fake news detection,” in Combating Online Hostile Posts in Regional Languages during Emergency Situation: First International Workshop, CONSTRAINT 2021, Collocated with AAAI 2021, Virtual Event, February 8, 2021, Revised Selected Papers 1, Springer, 2021, 153–163. https://doi.org/10.1007/978-3-030-73696-5_15
V. Dogra, A. Singh, S. Verma, Kavita, N. Z. Jhanjhi, and M. N. Talib, “Analyzing DistilBERT for sentiment classification of banking financial news,” in Intelligent Computing and Innovation on Data Science: Proceedings of ICTIDS 2021, Springer, 2021, 501–510. https://doi.org/10.1007/978-981-16-3153-5_53
J. Y. Khan, M. T. I. Khondaker, S. Afroz, G. Uddin, and A. Iqbal, “A benchmark study of machine learning models for online fake news detection,” Machine Learning with Applications, 4, 100032, 2021. https://doi.org/10.1016/j.mlwa.2021.100032
U. Saha et al., “Exploring public attitude towards children by leveraging emoji to track out sentiment using distil-BERT a fine-tuned model,” in International Conference on Image Processing and Capsule Networks, Springer, 2022, 332–346. https://doi.org/10.1007/978-3-031-12413-6_26
M. Abdullah, O. Altiti, and R. Obiedat, “Detecting propaganda techniques in english news articles using pre-trained transformers,” in 2022 13th International Conference on Information and Communication Systems (ICICS), IEEE, 2022, 301–308. DOI:10.1109/ICICS55353.2022.9811117
P. K. Verma, P. Agrawal, V. Madaan, and C. Gupta, “UCred: fusion of machine learning and deep learning methods for user credibility on social media,” Soc Netw Anal Min, 12)1:( 54, 2022. https://doi.org/10.1007/s13278-022-00880-1
H. Karande, R. Walambe, V. Benjamin, K. Kotecha, and T. S. Raghu, “Stance detection with BERT embeddings for credibility analysis of information on social media,” PeerJ Comput Sci, 7, e467, 2021. https://doi.org/10.7717/peerj-cs.467
J. Liu, T. Singhal, L. T. M. Blessing, K. L. Wood, and K. H. Lim, “Crisisbert: a robust transformer for crisis classification and contextual crisis embedding,” in Proceedings of the 32nd ACM conference on hypertext and social media, 2021, 133–141. https://doi.org/10.1145/3465336.3475117
F. Balouchzahi, H. L. Shashirekha, and G. Sidorov, “MUCIC at CheckThat! 2021: FaDo-Fake News Detection and Domain Identification using Transformers Ensembling.,” in CLEF (Working Notes), 2021, 455–464. https://doi.org/10.1007/978-3-030-85251-1_1
R. O. Ogundokun, M. O. Arowolo, S. Misra, and I. D. Oladipo, “Early detection of fake news from social media networks using computational intelligence approaches,” Combating fake news with computational intelligence techniques, 71–89, 2022. https://doi.org/10.1007/978-3-030-90087-8_4
S.-C. B. Lo, H.-P. Chan, J.-S. Lin, H. Li, M. T. Freedman, and S. K. Mun, “Artificial convolution neural network for medical image pattern recognition,” Neural networks, 8)7–8:( 1201–1214, 1995. https://doi.org/10.1016/0893-6080(95)00061-5
Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: analysis, applications, and prospects,” IEEE Trans Neural Netw Learn Syst, 33)12:( 6999–7019, 2021. DOI:10.1109/TNNLS.2021.3084827
J. Gu et al., “Recent advances in convolutional neural networks,” Pattern Recognit, 77, 354–377, 2018. https://doi.org/10.1016/j.patcog.2017.10.013
A. Patil and M. Rane, “Convolutional neural networks: an overview and its applications in pattern recognition,” Information and Communication Technology for Intelligent Systems: Proceedings of ICTIS 2020, Volume 1, 21–30, 2021. https://doi.org/10.1007/978-981-15-7078-0_3
Z. Xu, S. Qian, X. Ran, and J. Zhou, “Application of Deep Convolution Neural Network in Crack Identification,” Applied Artificial Intelligence, 36, )1( 2014188, 2022. https://doi.org/10.1080/08839514.2021.2014188
R. Shyam, “Convolutional neural network and its architectures,” Journal of Computer Technology & Applications, 12)2:( 6–14p, 2021. DOI:10.20944/preprints201811. 0546.v1
J. Yang and J. Li, “Application of deep convolution neural network,” in 2017 14th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), IEEE, 2017, 229–232. DOI:10.1109/ICCWAMTIP.2017.8301485
B. Gao and L. Pavel, “On the properties of the softmax function with application in game theory and reinforcement learning,” arXiv preprint arXiv:1704.00805, 2017. https://doi.org/10.48550/arXiv.1704.00805
K. Banerjee, R. R. Gupta, K. Vyas, and B. Mishra, “Exploring alternatives to softmax function,” arXiv preprint arXiv:2011.11538, 2020. https://doi.org/10.48550/arXiv.2011.11538
M. Hort, R. Moussa, and F. Sarro, “Multi-objective search for gender-fair and semantically correct word embeddings,” Appl Soft Comput, 133, 109916, 2023. https://doi.org/10.1016/j.asoc.2022.109916
N. Badri, F. Kboubi, and A. H. Chaibi, “Combining fasttext and glove word embedding for offensive and hate speech text detection,” Procedia Comput Sci, 207, 769–778, 2022. https://doi.org/10.1016/j.procs.2022.09.132
P. M. Brennan, J. J. M. Loan, N. Watson, P. M. Bhatt, and P. A. Bodkin, “Pre-operative obesity does not predict poorer symptom control and quality of life after lumbar disc surgery,” Br J Neurosurg, 31)6:( 682–687, 2017. https://doi.org/10.1080/02688697.2017.1354122
S. Hochreiter, “Long Short-term Memory,” Neural Computation MIT-Press, 1997. DOI:10.1162/neco.1997.9.8.1735
B. Gülmez, “Stock price prediction with optimized deep LSTM network with artificial rabbit’s optimization algorithm,” Expert Syst Appl,227, 120346, 2023. https://doi.org/10.1016/j.eswa.2023.120346
Q. Kang, E. J. Chen, Z.-C. Li, H.-B. Luo, and Y. Liu, “Attention-based LSTM predictive model for the attitude and position of shield machine in tunneling,” Underground Space, 13, 335–350, 2023. https://doi.org/10.1016/j.undsp.2023.05.006
J. Luo and Y. Gong, “Air pollutant prediction based on ARIMA-WOA-LSTM model,” Atmos Pollut Res, 14)6:( 101761, 2023. https://doi.org/10.1016/j.apr.2023.101761
S. R. Patra and H.-J. Chu, “Regional groundwater sequential forecasting using global and local LSTM models,” J Hydrol Reg Stud, 47, 101442, 2023. https://doi.org/10.1016/j.ejrh.2023.101442
V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,”. 2–6, 2019. DOI:10.48550/arXiv.1910.01108
B. Büyüköz, A. Hürriyetoğlu, and A. Özgür, “Analyzing ELMo and DistilBERT on socio-political news classification,” in Proceedings of the Workshop on Automated Extraction of Socio-political Events from News 2020, 2020, 9–18. DOI:10.32604/jqc.2022.026658
M. Abadeer, “Assessment of DistilBERT performance on named entity recognition task for the detection of protected health information and medical concepts,” in Proceedings of the 3rd clinical natural language processing workshop, 2020, 158–167. DOI: 10.18653/v1/2020.clinicalnlp-1.18
V. Sanh, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019. DOI:10.48550/arXiv.1910.01108
P. Singh, N. Singh, K. K. Singh, and A. Singh, “Diagnosing of disease using machine learning,” in Machine learning and the internet of medical things in healthcare, Elsevier, 2021, 89–111. https://doi.org/10.1016/B978-0-12-821229-5.00003-3
S. Reddi and G. V Eswar, “Fake news in social media recognition using Modified Long Short-Term Memory network,” in Security in IoT Social Networks, Elsevier, 2021, 205–227. https://doi.org/10.1016/B978-0-12-821599-9.00009-1
DOI:
https://doi.org/10.31449/inf.v49i25.7710Downloads
Published
How to Cite
Issue
Section
License
I assign to Informatica, An International Journal of Computing and Informatics ("Journal") the copyright in the manuscript identified above and any additional material (figures, tables, illustrations, software or other information intended for publication) submitted as part of or as a supplement to the manuscript ("Paper") in all forms and media throughout the world, in all languages, for the full term of copyright, effective when and if the article is accepted for publication. This transfer includes the right to reproduce and/or to distribute the Paper to other journals or digital libraries in electronic and online forms and systems.
I understand that I retain the rights to use the pre-prints, off-prints, accepted manuscript and published journal Paper for personal use, scholarly purposes and internal institutional use.
In certain cases, I can ask for retaining the publishing rights of the Paper. The Journal can permit or deny the request for publishing rights, to which I fully agree.
I declare that the submitted Paper is original, has been written by the stated authors and has not been published elsewhere nor is currently being considered for publication by any other journal and will not be submitted for such review while under review by this Journal. The Paper contains no material that violates proprietary rights of any other person or entity. I have obtained written permission from copyright owners for any excerpts from copyrighted works that are included and have credited the sources in my article. I have informed the co-author(s) of the terms of this publishing agreement.
Copyright © Slovenian Society Informatika
