BreastEnsemNet: Transformer and BiLSTM-Based Hybrid Ensemble Deep Learning for Mammogram Classification

Abstract

Breast cancer is still a leading cause of cancer death in women worldwide, supporting the requirement for accurate and timely diagnosis. Although deep learning models have obtained promising results for the automatic classification of mammograms, they are often limited by the need for efficient multi-scale feature extraction, spatial attention, and sequential dependency modeling. In this work, we present a hybrid ensemble deep learning framework, called BreastEnsemNet, which incorporates three complementary deep learning methodologies, including (i) deep hierarchical low-level to multi-scale feature extraction using VGG16, ResNet50 and InceptionV3, (ii) attention-based transformer detailed with spatial focus on well-relevant areas, and (iii) BiLSTM for capturing the sequential patterns and dependencies in the extracted features. There is no existing method that can automatically and efficiently combine models to achieve better fusion accuracy. The framework is trained and tested using the CBISDDSM mammogram dataset, where SMOTE is employed for class balancing, and various augmentation techniques are applied to facilitate generalization. BreastEnsemNet achieved better results with 98.79% accuracy, 97.9% precision, 98.4% recall, 98.1% F1-score, and an AUC-ROC of 99.2, outperforming multiple baseline models. The joint modeling of attention and sequences yielded a significant performance improvement for malignancy detection, resulting in a reduction in false negatives. These findings establish BreastEnsemNet's clinical utility as a practical, AI-based diagnostic aid for reliable and explainable breast cancer detection in mammograms.