Cross-Modal Attention GAN for Text-to-Artistic Image Generation

Abstract

Text and image are two very different data modalities. In the process of converting text-to-image, the essential difference between the two modalities leads to low R-value accuracy (a metric for semantic consistency) in generating the final image. In order to improve the relevance and artistry of the generated design images, a cross-modal attention-based method for generating artistic design images from text is investigated. A cross-modal attention-based generative adversarial network model (CMAGAN) is constructed to realize text-generated art and design images. The CMAGAN model is divided into three phases: in the initial cross-modal image generation phase, the pre-trained RNN is used to encode text descriptions, obtain sentence and word feature vectors, and generate initial cross-modal art and design images through the content-aware up-sampling module and channel-attention convolution module; In the initial image refinement stage, spatial and channel attention mechanisms are introduced to accurately match text and image features; in the image refinement stage, the image features are further attended to and fused using the secondary attention (AoA) mechanism to enhance the visual features; and under the effect of the integrated loss function, the semantically consistent, high-quality and richly-detailed art design images are obtained from the textual descriptions. Conduct experiments using datasets consisting of 6000 images each from Artstor and DeviantArt, two online art platforms. The ablation experiment showed that the complete model had the highest R-value accuracy (such as 0.86 for art, furniture, advertising, and graphics), the highest initial score (IS) (such as 3.65 for art), and the lowest Frechet Inception distance score (FID) (such as 20.4 for art) when generating art, furniture, advertising, and graphics design images. This indicates that the generated images have the strongest semantic consistency with the input text, are clearer and more diverse, and have the shortest distribution distance from real images. Compared with existing methods, the proposed method exhibits better scores in visual semantic similarity (VSS) across multiple sample sizes, with a more significant improvement and consistently maintaining a high level. The above results fully verify the advantages of our method in generating artistic design images from text.