A Deep Multimodal Retrieval Framework for Digital Libraries Using SE-ResNet-FCN, BERT, and Enhanced Hash Learning

Abstract

The current multimodal retrieval accuracy of online libraries is insufficient. To solve this problem, a multimodal retrieval model for digital libraries that integrates fully convolutional networks and hash learning is proposed in the proposed method. The research introduces a fully convolutional network and a bidirectional Transformer encoder to extract semantic features, and combines a residual neural network to deeply optimize the model, thereby enhancing the feature expression ability. In the hash learning stage, a triplet loss and contrastive learning loss optimization model is designed to further enhance cross-modal semantic alignment. This enables image-text multimodal retrieval. In the experiment, the model used was applied to the BookCoverDataset for verification. And it is combined with Latent semantic sparse hashing (LSSH) and Collective Matrix Factorization Hashing. CMFH, Supervised Matrix Factorization Hashing (SMFH), Discrete online cross-modal hashing the multimodal retrieval models of digital libraries constructed by DOCH were compared. The experimental results show that the average retrieval accuracy score of this model is up to 0.93, and the maximum mAP reaches 0.95, which is 0.20 higher than that of the comparison model. When the hash code is 256 bits, its average accuracy reaches 0.94. Compared with the baseline model, the study proposes that the model demonstrates stronger semantic association ability and feature compression efficiency in multimodal retrieval tasks, verifying the effectiveness of the fusion strategy of fully convolutional networks and hash learning. The model significantly enhances the accuracy and robustness of cross-modal retrieval through a deep semantic alignment mechanism, providing a feasible solution for efficient and precise image-text mutual inspection in the digital library environment.